Hybrid rendering system and hybrid rendering method

By determining and rendering user-interactive objects of the virtual environment on the client device and mixing them with the background scene frames received by the remote server, the problem of image quality degradation and delay in remote virtual reality is solved, and a better VR experience is achieved .

CN113781656BActive Publication Date: 2024-11-22HTC CORP
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202110640868.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-06-10
Filing Date
2021-06-09
Publication Date
2024-11-22
Estimated Expiration
2041-06-09

AI Technical Summary

Technical Problem

Remote virtual reality solutions lead to degradation in image quality and delays when streaming VR content, hindering user experience.

Method used

Using a hybrid rendering system and method, the client device determines that the user-interactive objects of the virtual environment, renders them by itself and mixes them with the background scene frames received by the remote server to provide high-quality visual content.

Benefits of technology

Through this method, the image quality and low latency in the VR experience are improved, and the immersion of users is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113781656B_ABST
    Figure CN113781656B_ABST
Patent Text Reader

Abstract

The present disclosure provides a hybrid rendering system and a hybrid rendering method. The hybrid rendering system includes a client device, which is configured to perform the following operations: determining at least one user-interactive object of a virtual environment; rendering at least one user-interactive object; receiving a background scene frame of the virtual environment; mixing the rendered at least one user-interactive object with the background scene frame into visual content of the virtual environment; and providing the visual content of the virtual environment. Therefore, the quality and latency of the rendered user-interactive object in the visual content can be better guaranteed, thereby improving the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to a rendering technology, and more particularly, to a hybrid rendering system and a hybrid rendering method. Background Art

[0002] Remote virtual reality (VR) solutions allow low-end VR devices to experience VR content of satisfactory quality by streaming from a remote high-efficiency graphics server, such as playing SteamVR content on a mobile VR device. However, in order to achieve smooth streaming, the remote server needs to compress / encode the rendered frames and transmit them to the VR device for decoding and display. This process usually results in image quality degradation and latency, both of which hinder the VR experience. Summary of the invention

[0003] Therefore, the present disclosure is directed to a hybrid rendering system and a hybrid rendering method, which can be used to solve the above technical problems.

[0004] In an embodiment of the present disclosure, the present disclosure provides a hybrid rendering system including a client device. The client device is configured to perform the following operations: determine at least one user-interactive object of a virtual environment; render at least one user-interactive object; receive a background scene frame of the virtual environment; blend the rendered at least one user-interactive object with the background scene frame into visual content of the virtual environment; and provide the visual content of the virtual environment.

[0005] In an embodiment of the present disclosure, the present disclosure provides a hybrid rendering method, which is applicable to a client device of a hybrid rendering system. The method includes: determining at least one user-interactive object of a virtual environment by the client device; rendering at least one user-interactive object by the client device; receiving a background scene frame of the virtual environment by the client device; mixing the rendered at least one user-interactive object and the background scene frame into visual content of the virtual environment by the client device; and providing the visual content of the virtual environment by the client device. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and together with the description serve to explain the principles of the invention.

[0007] Figure 1 A schematic diagram of a hybrid rendering system according to an embodiment of the present disclosure is shown.

[0008] Figure 2 A flow chart of a hybrid rendering method according to an embodiment of the present disclosure is shown.

[0009] Reference numerals list

[0010] 100: Hybrid Rendering System

[0011] 102: Remote server

[0012] 104: Client device

[0013] A1: Additional images

[0014] F1: Background scene frame

[0015] O1: User-interactive objects

[0016] S210, S220, S230, S240, S250: Steps

[0017] V1: Visual Content DETAILED DESCRIPTION

[0018] Reference will now be made in detail to the presently preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and description to refer to the same or like parts.

[0019] refer to Figure 1 , which depicts a schematic diagram of a hybrid rendering system according to an embodiment of the present disclosure. Figure 1 , the hybrid rendering system 100 includes a remote server 102 and a client device 104. In various embodiments, the client device 104 may be any device that can be used to run a VR service, such as a standalone head-mounted display (HMD), a computer device (such as a personal computer or the like), but the present disclosure is not limited thereto.

[0020] In an embodiment of the present disclosure, the remote server 102 connected to the client device 104 may be any device configured with high graphics processing capabilities. Roughly speaking, when the client device 104 runs a VR service to provide visual content to a user, the remote server 102 may help the client device 104 render some parts of the virtual environment of the VR service, and transmit the rendered parts to the client device 104. At the same time, the client device 104 may render some specific objects of the virtual environment by itself, and mix these rendered specific objects with the above-mentioned rendered parts received from the remote server 102 into visual content for providing / displaying to the user of the client device 104. A detailed discussion will be provided below.

[0021] See also Figure 2 , which depicts a flow chart of a hybrid rendering method according to an embodiment of the present disclosure. In this embodiment, Figure 2 The method can be Figure 1The client device 104 in the embodiment of the present invention is executed, and thus will be combined with Figure 1 The components shown in FIG. Figure 2 The details of each step are described in detail, but the present disclosure is not limited thereto.

[0022] First, in step S210 , the client device 104 may determine a user-interactable object O1 of the virtual environment. In various embodiments, the client device 104 may determine the user-interactable object O1 based on the teachings of at least one of the following embodiments.

[0023] In a first embodiment, the client device 104 may belong to a VR system that can provide VR services corresponding to a virtual environment. In one embodiment, the VR system may include an HMD connected to the client device 104. In other embodiments, the client device 104 may be an HMD. In these cases, the virtual environment may include a user representative object corresponding to the HMD, wherein the user representative object may be an object in the virtual environment corresponding to the user / HMD, but the present disclosure is not limited thereto. In one embodiment, the user representative object may move in the virtual environment in response to the movement of the HMD, but the present disclosure is not limited thereto.

[0024] In the first embodiment, the user-interactive object O1 may be a virtual object within a predetermined distance from the user-representative object. From another perspective, the client device 104 may obtain the distance between the user-representative object and each of the other objects in the virtual environment, and define those objects that are less than the predetermined distance from the user-representative object as the user-interactive object O1. In short, the client device 104 may determine the virtual object near the user-representative object as the user-interactive object O1, but the present disclosure is not limited thereto.

[0025] In a second embodiment, the VR system may further include a handheld controller connected to the client device 104, and the virtual environment may include a controller representative object corresponding to the handheld controller. In the second embodiment, the user-interactive object O1 may include a virtual object within a predetermined distance from the controller representative object. From another perspective, the client device 104 may obtain the distance between the controller representative object and each of the other objects in the virtual environment, and define those objects that are less than a predetermined distance from the controller representative object as the user-interactive object O1. In short, the client device 104 may determine the virtual object near the controller representative object as the user-interactive object O1, but the present disclosure is not limited to this.

[0026] In the third embodiment, the client device 104 may define the controller representative object as the user interactable object O1 , but the present disclosure is not limited thereto.

[0027] In a fourth embodiment, the controller representative object can move with the handheld controller in the virtual environment. That is, when the user physically moves the handheld controller while immersed in the VR service, the controller representative object will move in the virtual environment in response to the movement of the handheld controller. In this case, the client device 104 can predict the movement path of the controller representative object within a predetermined time. In various embodiments, the client device 104 can predict the movement path of the controller representative object based on any conventional movement prediction technology. For example, the client device 104 can predict the movement path of the controller representative object based on (for example) historical movement speed, the rotation angle of the controller representative object, but the present disclosure is not limited to this. In one embodiment, since the controller representative object can move in response to the movement of the handheld controller, the client device 104 can predict the movement path of the controller representative object based on (for example) historical movement speed, the rotation angle of the handheld controller, but the present disclosure is not limited to this.

[0028] Next, the client device 104 may obtain one or more virtual objects on the moving path of the controller representative object within a predetermined time, and define the one or more virtual objects on the moving path as the user interactable object O1.

[0029] In a fifth embodiment, the client device 104 may predict the movement path of the user representative object within a predetermined time. For example, the client device 104 may predict the movement path of the user representative object based on, for example, historical movement speed, rotation angle of the user representative object, but the present disclosure is not limited thereto. In one embodiment, since the user representative object may move in response to the movement of the HMD, the client device 104 may predict the movement path of the user representative object based on, for example, historical movement speed, rotation angle of the HMD, but the present disclosure is not limited thereto.

[0030] Next, the client device 104 may obtain one or more virtual objects on the moving path of the user representative object within a predetermined time, and define the one or more virtual objects as the user interactable object O1.

[0031] In other embodiments, the client device 104 may determine one or more user-interactive objects O1 based on at least one of the teachings in the first to fifth embodiments. For example, the client device 104 may determine a virtual object near a user representative object and / or a controller representative object as one or more user-interactive objects O1. Additionally or alternatively, the client device 104 may determine a virtual object on a predicted movement path of a user representative object and / or a controller representative object as one or more user-interactive objects O1. Additionally or alternatively, the client device 104 may further determine a controller representative object as one of the one or more user-interactive objects O1, but the present disclosure is not limited thereto.

[0032] In other embodiments, the remote server 102 may be configured to determine the user interactable object O1 based on the principles taught in the first to fifth embodiments, and accordingly notify the client device 104 so that the client device 104 determines the user interactable object O1, but the present disclosure is not limited thereto.

[0033] For the user-interactive object O1, in step S220, the client device 104 may render the user-interactive object O1. From another perspective, one or more user-interactive objects O1 may be roughly understood as virtual objects that are near the user representative object and / or are more likely to interact with the user. For such virtual objects, the client device 104 may decide to render these virtual objects by itself to better ensure quality and latency, so that the user can have a better VR experience, but the present disclosure is not limited thereto.

[0034] Next, in step S230, the client device 104 may receive a background scene frame F1 of the virtual environment. In one embodiment, the remote server 102 may determine the background portion of the virtual environment, render the background portion as a background scene frame F1, and transmit the background scene frame F1 to the client device 104, but the present disclosure is not limited thereto. In one embodiment, the remote server 102 may determine the user-interactive object O1 based on the same principle used by the client device 104 to determine the user-interactive object O1. Next, the remote server 102 may determine the portion other than the user-interactive object O1 as the background portion, but the present disclosure is not limited thereto. In other embodiments, the client device 104 may notify the remote server 102 of the user-interactive object O1 accordingly, and the remote server 102 may accordingly determine the portion other than the user-interactive object O1 as the background portion.

[0035] Generally speaking, the content of the background part is not as visually important as the user-interactive object O1. Therefore, the remote server 102 can help the client device 104 render the background part as the background scene frame F1 to reduce the load of the client device 104, but the present disclosure is not limited thereto.

[0036] In step S240 , the client device 104 may mix the rendered user-interactive object O1 with the background scene frame F1 into the visual content V1 of the virtual environment. As mentioned above, since the user-interactive object O1 is rendered by the client device 104 , the user-interactive object O1 may provide better quality to the user with low latency.

[0037] In other embodiments, the remote server 102 may determine a text-rich object in the virtual environment and render the rich text object. In some embodiments, the rich text object may include some specific objects that are not part of the background, such as a VR menu containing a certain amount of rich text, but the present disclosure is not limited thereto. In this case, the rich text object is rendered as an additional image A1 and the additional image A1 is transmitted to the client device 104. Therefore, the client device 104 may further mix the additional image A1 with the rendered user-interactive object O1 and the background scene frame F1 into the visual content V1 of the virtual environment.

[0038] In some embodiments, the remote server 102 may transmit the additional image A1 and the background scene frame F1 by using different streams with different frame rates. For example, since the additional image A1 may not be as important as the background scene frame F1, the remote server 102 may transmit the additional image A1 by using a stream with a lower frame rate, and transmit the background scene frame F1 by using a stream with a higher frame rate, but the present disclosure is not limited thereto.

[0039] For the visual content V1, in step S250, the client device 104 may provide the visual content V1 of the virtual environment. In one embodiment, the client device 104 may display the visual content V1 for the user to view. Because the rendered user-interactive object O1 can be presented with high quality and low latency, the user can have a better VR experience.

[0040] In some embodiments, if the networking delay of the retransmission is too high, the background scene frame F1 may be damaged due to packet loss. In this case, the client device 104 may determine whether the frame portion of the background scene frame F1 has a decoding error. In one embodiment, in response to determining that a decoding error occurs at the first frame portion of the background scene frame F1, the client device 104 may obtain a second frame portion of the previous background scene frame, wherein the second frame portion corresponds to the first frame portion. In one embodiment, assuming that the background scene frame F1 is the i-th frame from the remote server 102, the previous background scene frame may be the (ik)-th frame from the remote server 102, where i is an index and k may be any integer (e.g., 1), but the present disclosure is not limited thereto.

[0041] In one embodiment, the relative position between the first frame portion and the background scene frame F1 may be the same as the relative position between the second frame portion and the previous background scene frame, but the present disclosure is not limited thereto.

[0042] Next, the client device 104 may reproject the second frame portion into a third frame portion and replace the first frame portion with the third frame portion. In one embodiment, the client device 104 may reproject the second frame portion into a third frame portion based on asynchronous time warp (ATW), but the present disclosure is not limited thereto. Therefore, after the client device 104 mixes the user-interactive object O1 with the background scene frame F1 (whose first frame portion has been replaced by the third frame portion) into visual content V1, the client device 104 may provide the user with visual content V1 without decoding errors, so that the user experience will not be affected by the first frame portion with decoding errors, but the present disclosure is not limited thereto.

[0043] In summary, the embodiments of the present disclosure can determine the user-interactive objects in the virtual embodiment based on one or more principles taught in the first to fifth embodiments, and allow the client device to render the user-interactive objects by itself. After receiving the background scene frame from the remote server, the client device can mix the rendered user-interactive objects with the background scene frame to generate visual content for the user to view. Since the user-interactive objects are rendered by the client device, the quality and latency of the rendered user-interactive objects in the visual content can be better guaranteed, thereby improving the user experience.

[0044] It will be apparent to those skilled in the art that various modifications and variations may be made to the structure of the present invention without departing from the scope or spirit of the present invention. In view of the foregoing, it is desired that the present invention encompasses modifications and variations of the present invention, provided that the modifications and variations fall within the scope of the appended claims and their equivalents.

Claims

1. A client device, characterized in that: Configured to do the following: determining at least one user-interactable object of the virtual environment; Rendering the at least one user-interactable object; Receiving a background scene frame of the virtual environment; blending the rendered at least one user-interactable object with the background scene frame into visual content of the virtual environment; as well as Providing the visual content of the virtual environment, wherein the client device includes a head mounted display of a virtual reality system, or includes a computing device connected to the head mounted display, wherein the virtual reality system provides a virtual reality service corresponding to the virtual environment, wherein the virtual reality system further includes at least one handheld controller connected to the client device, and the virtual environment includes a user representative object corresponding to the head mounted display and at least one controller representative object corresponding to the at least one handheld controller, the at least one controller representative object moves with the at least one handheld controller, and the client device is further configured to perform the following operations: Predicting the movement path of each of the controller representative objects or the user representative objects within a predetermined time; Obtaining at least one virtual object on the moving path of each controller representative object or the user representative object within the predetermined time; as well as The at least one virtual object is defined as the at least one user-interactive object.

2. A hybrid rendering method, applicable to a client device of a hybrid rendering system, characterized in that: The method comprises: determining, by the client device, at least one user-interactable object of the virtual environment; rendering, by the client device, the at least one user-interactable object; Receiving, by the client device, a background scene frame of the virtual environment; blending, by the client device, the rendered at least one user-interactable object with the background scene frame into visual content of the virtual environment; and The visual content of the virtual environment is provided by the client device, wherein the client device includes a head mounted display of a virtual reality system, or includes a computing device connected to the head mounted display, wherein the virtual reality system provides a virtual reality service corresponding to the virtual environment, wherein the virtual reality system further includes at least one handheld controller connected to the client device, and the virtual environment includes a user representative object corresponding to the head mounted display and at least one controller representative object corresponding to the at least one handheld controller, and the at least one controller representative object moves with the at least one handheld controller, and the method further includes: The client device predicts a moving path of each of the controller representative objects or the user representative objects within a predetermined time; obtaining, by the client device, at least one virtual object on the moving path of each of the controller representative objects or the user representative objects within the predetermined time; and The at least one virtual object is defined by the client device as the at least one user-interactable object. 3 . The hybrid rendering method according to claim 2 , wherein the at least one user-interactive object further comprises the at least one controller representative object corresponding to the at least one handheld controller.

4. The hybrid rendering method according to claim 2, wherein the hybrid rendering system further comprises a remote server connected to the client device, and the method further comprises: determining, by the remote server, a background portion of the virtual environment; The remote server renders the background portion into the background scene frame; Determining, by the remote server, a rich text object in the virtual environment; rendering, by the remote server, the rich text object as an additional image; transmitting, by the remote server, the background scene frame and the additional image to the client device, wherein the additional image and the background scene frame are transmitted by using different streams having different frame rates, respectively; as well as The additional image is blended, by the client device, with the rendered at least one user interactable object and the background scene frame into the visual content of the virtual environment.

5. The hybrid rendering method according to claim 2, further comprising: In response to determining that a decoding error occurs at a first frame portion of the background scene frame, obtaining, by the client device, a second frame portion of a previous background scene frame, wherein the second frame portion corresponds to the first frame portion; as well as The second frame portion is reprojected by the client device as a third frame portion and the first frame portion is replaced with the third frame portion.

Citation Information

Patent Citations

  • Collaborative rendering method for virtual reality

    CN107274469A

  • Remote rendering for virtual images

    CN108351691A

  • Avatars in virtual environments

    CN108697935A

  • Augmented reality and virtual reality feedback enhancement system, apparatus and method

    CN108733339A

  • Dragging virtual elements of an augmented and / or virtual reality environment

    US20180136743A1