Rendering methods, electronic devices and systems

Abstract

This invention provides a rendering method, electronic device, and system, relating to the field of rendering technology. The method, applied to an XR device, includes: receiving rendering data acquired by the XR device; performing pre-rendering processing on the rendering data using a native Monado runtime, and submitting the obtained rendering scene data to a streaming XR application; sending the rendering scene data to a computing device via the streaming XR application, causing the computing device to perform rendering processing, and sending the obtained rendering result back to the XR device; receiving the rendering result sent by the computing device via the streaming XR application, and submitting the rendering result to the native Monado runtime; and displaying the rendering result via the native Monado runtime. This invention solves the problems of poor performance, short battery life, and poor wearing comfort in existing XR headsets.

Description

Technical Field

[0001] This invention relates to the field of rendering technology, and more particularly to a rendering method, electronic device, and system. Background Technology

[0002] Extended Reality (XR) technology, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), relies on high-performance, low-latency real-time 3D rendering capabilities to create realistic and immersive environments. Currently, mainstream consumer XR devices generally adopt a standalone head-mounted display (HMD) design, highly integrating computing units, display units, sensors, and batteries into the user-worn head-mounted device. While this design improves portability and ease of deployment, it introduces significant and irreconcilable contradictions in rendering performance, battery life, and thermal management, becoming a key bottleneck restricting the upgrading of the XR experience.

[0003] (1) Performance vs. portability: Due to the internal space constraints and heat dissipation requirements of all-in-one head-mounted devices, they cannot use high-performance core processors, resulting in an insurmountable technical gap between rendering capabilities and portability.

[0004] (2) Poor battery life: In order to make users comfortable to wear, the all-in-one XR headset has strict weight restrictions, so it is impossible to use a large-capacity battery, which greatly limits the battery life of XR devices;

[0005] (3) Heat dissipation bottleneck: High-performance rendering generates a lot of heat. In a highly integrated, enclosed, and close-fitting all-in-one XR headset, users will feel uncomfortable when wearing it.

[0006] Therefore, how to resolve the core contradictions in performance, battery life, and wearing comfort of XR headsets at the current stage is an urgent problem to be solved. Summary of the Invention

[0007] This invention provides a rendering method, electronic device, and system to solve the problems of poor performance, short battery life, and poor wearing comfort of existing XR headsets.

[0008] This invention provides a rendering method applied to XR devices, comprising:

[0009] Receive the data to be rendered acquired by the XR device;

[0010] The rendering preprocessing of the data to be rendered is performed by the native Monado runtime, and the resulting rendering scene data is submitted to the streaming XR application.

[0011] The streaming XR application sends the rendering scene data to the computing device, which then performs rendering processing and sends the resulting rendering result to the XR device.

[0012] The streaming XR application receives the rendering results sent by the computing device and submits the rendering results to the native Monado runtime.

[0013] The rendering results are displayed using the native Monado runtime.

[0014] According to a rendering method provided by the present invention, the computing device includes a streaming Monado runtime and a native XR application; wherein, the streaming Monado runtime is used to receive rendering scene data sent by the streaming XR application;

[0015] The native XR application is used to render the scene data to obtain the rendering result.

[0016] According to a rendering method provided by the present invention, the step of sending the rendered scene data to a computing device through the streaming XR application includes:

[0017] The streaming XR application sends the rendered scene data to the computing device via the WebRTC protocol.

[0018] According to a rendering method provided by the present invention, before performing pre-rendering processing on the data to be rendered using the native Monado runtime, the method further includes:

[0019] Get the current application;

[0020] The pre-rendering processing of the data to be rendered using the native Monado runtime includes:

[0021] Based on the current application, determine the target processing method;

[0022] The native Monado runtime performs pre-rendering processing on the data to be rendered according to the target processing method.

[0023] This invention provides a rendering method applied to a computing device, comprising:

[0024] The system receives rendering scene data sent by the XR device through the Monado streaming runtime and submits the rendering scene data to the native XR application.

[0025] The rendering scene data is rendered using the native XR application to obtain the rendering result, and the rendering result is submitted to the streaming Monado runtime.

[0026] The rendering result is sent to the XR device via the Monado streaming runtime, so that the XR device can display the rendering result.

[0027] According to a rendering method provided by the present invention, the XR device includes a native Monado runtime and a streaming XR application; wherein, the native Monado runtime is used to perform pre-rendering processing on the data to be rendered to obtain rendering scene data;

[0028] The streaming XR application is used to send the rendered scene data to the computing device, and also to receive the rendering results sent by the computing device and submit them to the native Monado runtime.

[0029] According to a rendering method provided by the present invention, the step of sending the rendering result to the XR device through the Monado streaming runtime includes:

[0030] The rendering results are sent to the XR device via the WebRTC protocol during the Monado streaming runtime.

[0031] According to a rendering method provided by the present invention, the step of rendering the scene data through a native XR application to obtain a rendering result includes:

[0032] Obtain the current application and determine the data update scenario based on the current application;

[0033] The native XR application renders the scene data based on the updated scene data to obtain the rendering result.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the rendering method as described in any of the preceding claims.

[0035] The present invention also provides a rendering system, including an XR device and a computing device, wherein:

[0036] The XR device is used to receive the data to be rendered acquired by the XR device, perform pre-rendering processing on the data to be rendered through the native Monado runtime to obtain rendering scene data, and send the rendering scene data to the computing device through a streaming XR application.

[0037] The computing device is configured to receive the rendering scene data through the Monado streaming runtime, perform rendering processing on the rendering scene data through a native XR application to obtain a rendering result, and send the rendering result to the XR device through the Monado streaming runtime.

[0038] The XR device is used to receive the rendering results through the streaming XR application and display the rendering results through the native Monado runtime.

[0039] The rendering method, electronic device, and system provided by this invention acquire data to be rendered through an XR device, perform pre-rendering processing on the data using the native Monado runtime, and submit the obtained rendering scene data to a streaming XR application. The streaming XR application then sends the rendering scene data to a computing device, which performs rendering processing and sends the resulting rendering result back to the XR device. Next, the streaming XR application receives the rendering result from the computing device and submits it to the native Monado runtime. Finally, the native Monado runtime displays the rendering result. This invention, through secondary development of the Monado open-source framework, implements a cross-platform OpenXR runtime framework with remote rendering capabilities. Specifically, without modifying the native XR application and the native Monado runtime, it introduces additional customized streaming XR applications and streaming Monado runtimes, separating the native XR application to run on the computing device and the native Monado runtime to run on the XR device. In other words, XR devices include a native Monado runtime and an integrated streaming XR application. The XR device uses the native Monado runtime for pre-rendering processing and then sends the processed rendering scene data to the computing device via the streaming XR application for rendering. By employing a high-performance computing device for rendering, this approach achieves rendering quality unattainable by traditional all-in-one XR devices, significantly enhancing the user's immersive experience. Furthermore, by offloading high-power rendering tasks to the computing device and separating the rendering hardware from the XR device, it allows for longer battery life with a smaller capacity battery. It also effectively reduces the size and heat generation of the XR device, making it more comfortable for users to wear. Attached Figure Description

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a diagram of the stand-alone rendering framework for existing XR devices;

[0042] Figure 2 This is a system architecture diagram of the rendering system provided by the present invention;

[0043] Figure 3 This is one of the flowcharts illustrating the rendering method provided by the present invention;

[0044] Figure 4 This is the second flowchart illustrating the rendering method provided by the present invention;

[0045] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention;

[0046] Wherein, the accompanying drawings are marked as follows:

[0047] 01: Native XR application; 02: Native Monado runtime; 03: Streaming XR application; 04: Streaming Monado runtime. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] Extended Reality (XR) technology, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), relies on high-performance, low-latency real-time 3D rendering capabilities to create realistic and immersive environments. Currently, mainstream consumer XR devices generally adopt a standalone head-mounted display (HMD) design, highly integrating computing units, display units, sensors, and batteries into the user-worn head-mounted device. While this design improves portability and ease of deployment, it introduces significant and irreconcilable contradictions in rendering performance, battery life, and thermal management, becoming a key bottleneck restricting the upgrading of the XR experience.

[0050] (1) Performance vs. portability: Due to the internal space constraints and heat dissipation requirements of all-in-one head-mounted devices, they cannot use high-performance core processors, resulting in an insurmountable technical gap between rendering capabilities and portability.

[0051] (2) Poor battery life: In order to make users comfortable to wear, the all-in-one XR headset has strict weight restrictions, so it is impossible to use a large-capacity battery, which greatly limits the battery life of XR devices;

[0052] (3) Heat dissipation bottleneck: High-performance rendering generates a lot of heat. In a highly integrated, enclosed, and close-fitting all-in-one XR headset, users will feel uncomfortable when wearing it.

[0053] Therefore, how to resolve the core contradictions in performance, battery life, and wearing comfort of XR headsets at the current stage is an urgent problem to be solved.

[0054] Furthermore, to address the aforementioned issues, existing technologies have proposed solutions using external computing units, such as the wireless streaming of HTC Vive (a VR headset jointly developed by HTC and Valve). However, this solution essentially binds the all-in-one XR headset to a fixed external computing unit (usually the user's PC), and cannot be easily replaced with computing units of different characteristics (such as portable smartphones) according to individual needs. Therefore, this solution only alleviates part of the computing power problem and does not truly solve the modular expansion requirement.

[0055] Based on the above problems, this invention proposes a rendering method, electronic device, and system, which are described below in conjunction with... Figure 1-Figure 5 Describe it.

[0056] Figure 1 This is a diagram of the stand-alone rendering framework for existing XR devices. For example... Figure 1 As shown, existing XR devices use an independent XR rendering framework. After obtaining the data to be rendered, the XR device performs pre-rendering processing on the data through the native Monado runtime 02, and then sends the processed rendering scene data to the native XR application 01. The native XR application 01 performs rendering processing on the rendering scene data to obtain the rendering result, and the native Monado runtime 02 displays the rendering result.

[0057] Based on the above Figure 1 This invention, without modifying the native XR application and the native Monado runtime, introduces additional customized streaming XR application 03 and streaming Monado runtime 04, thereby separating the native XR application 01 and the native Monado runtime 02 for execution on the computing device and XR device, respectively. For details, see... Figure 2 , Figure 2 This is a system architecture diagram of the rendering system provided by the present invention. (For example...) Figure 2 As shown, the rendering system includes an extended reality (XR) device 10 and a computing device 20.

[0058] The XR device 10 includes, but is not limited to, VR (Virtual Reality) devices, AR (Augmented Reality) devices, and MR (Mixed Reality) devices, used to execute the rendering method of the present invention. The computing device 20 can be a server, cloud server, PC (Personal Computer), smartphone, tablet computer, or other devices.

[0059] In this rendering system:

[0060] The XR device 10 is used to receive the data to be rendered acquired by the XR device, perform pre-rendering processing on the data to be rendered through the native Monado runtime to obtain rendering scene data, and send the rendering scene data to the computing device through a streaming XR application.

[0061] The computing device 20 is used to receive the rendering scene data through the Monado streaming runtime, perform rendering processing on the rendering scene data through a native XR application to obtain a rendering result, and send the rendering result to the XR device through the Monado streaming runtime.

[0062] The XR device 10 is used to receive the rendering result through the streaming XR application and display the rendering result through the native Monado runtime.

[0063] In this embodiment, the XR device acquires the data to be rendered, performs pre-rendering processing on the data using the native Monado runtime to obtain the rendered scene data, and submits it to the streaming XR application. Then, the streaming XR application sends the rendered scene data to the computing device. The data to be rendered includes, but is not limited to: acceleration, angular velocity, magnetic field strength, magnetic field direction, scene image, scene depth image, location data, radar data, voice data, and push notifications. The native Monado runtime refers to the Monado runtime originally used in the XR device. A streaming XR application refers to an application mode that uses streaming technology to transmit XR content (such as rendered scene data, rendered results, etc.) between the XR device and the computing device, so that the rendering task is executed on the computing device, and the rendered results are displayed and interacted with on the XR device.

[0064] After acquiring the rendering scene data through the streaming Monado runtime, the computing device submits it to the native XR application. The native XR application then processes the rendering scene data to obtain the rendering result, which is subsequently submitted to the streaming Monado runtime. The streaming Monado runtime refers to a modified or extended version of the open-source Monado runtime, enabling it to transmit the rendering results of the XR application running on the computing device to the XR device in real time, and to receive the data to be rendered from the XR device to the computing device. Native XR applications refer to various practical applications that originally utilize XR technology.

[0065] Next, the computing device sends the rendering results to the XR device via the streaming Monado runtime.

[0066] Finally, the XR device receives the rendering results through the streaming XR application and submits them to the native Monado runtime, which then displays the rendering results.

[0067] It should be understood that the rendering system described in this invention corresponds to the rendering method described below. That is, the specific processing procedures of the XR device and the computing device can be referred to the following embodiments, which will not be repeated here.

[0068] This invention provides a rendering system comprising an XR device and a computing device. Compared to traditional XR devices that use a standalone rendering framework with a native Monado runtime and native XR applications, this invention implements a cross-platform OpenXR runtime framework with remote rendering capabilities through secondary development of the Monado open-source framework. Specifically, without modifying the native XR application and native Monado runtime, additional customized streaming XR applications and streaming Monado runtimes are introduced. The native XR application runs on the computing device, and the native Monado runtime runs on the XR device. Correspondingly, the streaming XR application is introduced into the XR device, and the streaming Monado runtime is introduced into the computing device. The XR device performs pre-rendering processing using the native Monado runtime and sends the processed rendering scene data to the computing device via the streaming XR application for rendering. By employing a high-performance computing device for rendering, this method achieves rendering quality unattainable by traditional standalone XR devices, significantly enhancing the user's immersive experience. Meanwhile, by offloading high-power rendering tasks to computing devices and separating the rendering hardware from XR devices, XR devices can achieve longer battery life with smaller capacity batteries. This also effectively reduces the size and heat generation of XR devices, making them more comfortable for users to wear.

[0069] Furthermore, since the primary functions of the streaming XR application and the streaming Monado runtime in this invention are to transmit rendering scene data and rendering results, the actual XR experience still originates from the native XR application and the native Monado runtime. That is, this invention retains the program execution logic for the combined use of the XR application and the Monado runtime in both computing devices and XR devices. Therefore, the technical solution of this invention has the characteristics of high adaptability and high scalability.

[0070] Figure 3 This is one of the flowcharts illustrating the rendering method provided by the present invention, such as... Figure 3 As shown, the rendering method includes steps S110, S120, S130, S140 and S150.

[0071] Step S110: Receive the data to be rendered obtained by the XR device.

[0072] In this embodiment, the rendering method is applied to an XR device.

[0073] XR devices refer to wearable or portable devices that integrate virtual and real environments through hardware and software technologies to enable human-computer interaction. XR devices include, but are not limited to, VR devices, AR devices, and MR devices. VR devices use computer technology to simulate and generate a three-dimensional virtual space, allowing users to immerse themselves and interact with it, gaining a truly immersive experience. AR devices use technology to merge virtual information with the real world, overlaying it onto real-world scenes in real time to enhance sensory experience. MR devices mix the real and virtual worlds to create new visual environments that simultaneously contain physical entities and virtual information, allowing users to interact with these entities and information in real time. These XR devices can be head-mounted XR devices, such as AR glasses and VR glasses.

[0074] Here, the data to be rendered can be acquired through the sensors of the XR device, including but not limited to: attitude sensors, image sensors, GPS (Global Positioning System) sensors, LiDAR, and microphones. Attitude sensors include, but are not limited to, IMU (Inertial Measurement Unit), gyroscopes, accelerometers, and magnetometers, while image sensors include, but are not limited to, cameras, depth cameras, and eye-tracking sensors. The data to be rendered can also be obtained through applications on the XR device or computing device, including push notification applications such as calendars, schedule management apps, and health management apps. Correspondingly, the data to be rendered includes, but is not limited to: acceleration, angular velocity, magnetic field strength, magnetic field direction, scene images, scene depth images, location data, radar data, voice data, and push notification information.

[0075] One implementation approach is to first acquire the current application data, and then, based on the application type, acquire the corresponding data to be rendered. For example, if the current application is a navigation application, the acquired data to be rendered may include, but is not limited to, acceleration, angular velocity, scene images, scene depth images, location data, and radar data. As another example, if the current application is a translation application, the acquired data to be rendered may be voice data and / or scene images.

[0076] Step S120: Perform pre-rendering processing on the data to be rendered using the native Monado runtime, and submit the resulting rendering scene data to the streaming XR application.

[0077] XR devices include the native Monado runtime and streaming XR applications.

[0078] The Monado runtime is an open-source OpenXR runtime designed to provide immersive experiences. The Monado runtime refers to the runtime environment used by XR devices. Here, the native Monado runtime refers to the Monado runtime originally used in XR devices.

[0079] Streaming is a technology that transmits digital audio and video content in real time over the internet, allowing users to play it simultaneously during download without waiting for the complete file to download. XR applications refer to various practical applications using XR technology, such as navigation and translation applications using augmented reality displays. Streaming XR applications specifically refer to an application model that uses streaming technology to transmit XR content (such as rendered scene data and rendering results) between XR devices and computing devices, so that the rendering task is executed on the computing device, and the rendering results are then displayed and interacted with on the XR device.

[0080] The native Monado runtime performs pre-processing on the data to be rendered to obtain the rendering scene data. The specific processing method can be determined based on the current application.

[0081] Specifically, the current application can be obtained, the target processing method can be determined based on the current application, and then the native Monado runtime can perform pre-processing on the data to be rendered according to the target processing method. For example, if the current application is a translation application and the corresponding data to be rendered is voice data, the target processing method can be determined to be speech recognition processing. Then, the native Monado runtime can perform speech recognition processing on the data to be rendered to obtain the speech recognition result.

[0082] After obtaining the rendering scene data, the obtained rendering scene data is submitted to the streaming XR application.

[0083] As one implementation, streaming XR applications can obtain these rendering scene data from the native Monado runtime through the standard OpenXR (a programmable interface for XR applications) interface.

[0084] Step S130: The rendering scene data is sent to the computing device through the streaming XR application, so that the computing device performs rendering processing and sends the obtained rendering result to the XR device.

[0085] The XR application streams the rendering scene data to the computing device. Conversely, after receiving the rendering scene data, the computing device processes it to obtain the rendering result, which is then sent to the XR device.

[0086] As one implementation method, streaming XR applications can send rendered scene data to computing devices via wired transmission.

[0087] As another implementation method, streaming XR applications can send rendered scene data to computing devices via Wi-Fi (Wireless Fidelity) communication.

[0088] As another implementation, streaming XR applications can send rendered scene data to computing devices via the WebRTC (Web Real-Time Communications) protocol. WebRTC is a real-time communication technology that allows web applications or websites to establish peer-to-peer connections between XR devices and computing devices without the aid of an intermediary, enabling the transmission of rendered scene data, rendering results, or other arbitrary data.

[0089] Step S140: Receive the rendering result sent by the computing device through the streaming XR application, and submit the rendering result to the native Monado runtime.

[0090] XR devices can receive rendering results sent by computing devices through streaming XR applications, and then submit the rendering results to the native Monado runtime.

[0091] Specifically, the rendering results can be submitted to the native Monado runtime via the standard OpenXR interface.

[0092] Step S150: Display the rendering result using the native Monado runtime.

[0093] After receiving the rendering results, the native Monado runtime displays the rendering results.

[0094] The rendering method provided in this invention involves acquiring data to be rendered through an XR device, performing pre-rendering processing on the data using the native Monado runtime, and submitting the obtained rendering scene data to a streaming XR application. The streaming XR application then sends the rendering scene data to a computing device, which performs rendering processing and sends the resulting rendering result back to the XR device. Next, the streaming XR application receives the rendering result from the computing device and submits it to the native Monado runtime. Finally, the native Monado runtime displays the rendering result. This invention, through secondary development of the Monado open-source framework, implements a cross-platform OpenXR runtime framework with remote rendering capabilities. Specifically, without modifying the native XR application and the native Monado runtime, it introduces additional customized streaming XR applications and streaming Monado runtimes, separating the native XR application to run on the computing device and the native Monado runtime to run on the XR device. In other words, XR devices include a native Monado runtime and an integrated streaming XR application. The XR device uses the native Monado runtime for pre-rendering processing and then sends the processed rendering scene data to the computing device via the streaming XR application for rendering. By employing a high-performance computing device for rendering, this approach achieves rendering quality unattainable by traditional all-in-one XR devices, significantly enhancing the user's immersive experience. Furthermore, by offloading high-power rendering tasks to the computing device and separating the rendering hardware from the XR device, it allows for longer battery life with a smaller capacity battery. It also effectively reduces the size and heat generation of the XR device, making it more comfortable for users to wear.

[0095] In one embodiment, the computing device includes a streaming Monado runtime and a native XR application; wherein the streaming Monado runtime is used to receive rendering scene data sent by the streaming XR application; and the native XR application is used to render the rendering scene data to obtain a rendering result.

[0096] In this embodiment, the streaming Monado runtime refers to a modified or extended version of the open-source Monado runtime, enabling it to transmit the rendering results of XR applications running on the computing device to the XR device in real time, and to receive the data to be rendered transmitted from the XR device to the computing device. Native XR applications refer to various practical applications that originally used XR technology.

[0097] After receiving rendering scene data from a streaming XR application via the streaming Monado runtime, the computing device retrieves the rendering scene data from the streaming Monado runtime through the native XR application. Specifically, it can retrieve the rendering scene data from the streaming Monado runtime via the standard OpenXR interface. Then, it performs rendering processing on the rendering scene data to obtain the rendering result, and finally, it returns the rendering result to the streaming Monado runtime of the XR device via the standard OpenXR interface.

[0098] In this embodiment, without modifying the native XR application and native Monado runtime, additional customized streaming XR applications and streaming Monado runtimes are introduced. The native XR application runs on the computing device, and the native Monado runtime runs on the XR device; correspondingly, the streaming Monado runtime is introduced into the computing device. That is, the computing device includes both the streaming Monado runtime and the native XR application. Rendering is performed by the computing device and then sent to the XR device. By using a high-performance computing device for rendering, rendering quality unattainable by traditional all-in-one XR devices can be achieved, significantly enhancing the user's immersive experience. Simultaneously, this allows the XR device to achieve longer battery life with a smaller capacity battery, and effectively reduces the size and heat generation of the XR device, making it more comfortable for users to wear.

[0099] In one embodiment, step S130 includes:

[0100] The streaming XR application sends the rendered scene data to the computing device via the WebRTC protocol.

[0101] Here, WebRTC is a real-time communication technology that allows web applications or websites to establish point-to-point connections between XR devices and computing devices without the need for an intermediary, enabling the transmission of data to be rendered or other arbitrary data. WebRTC can run in environments such as Web, Android, iOS, Windows, macOS, and Linux, thus allowing XR devices to connect to different types of computing devices, such as PCs, cloud servers, and smartphones.

[0102] In this embodiment, by employing the WebRTC protocol as the transmission protocol for rendering scene data, low-latency performance requirements can be achieved. This significantly improves the user experience, especially for real-time scenarios. Furthermore, since the primary function of the streaming XR application and streaming Monado runtime in this invention is transmission, both the computing device and the XR device retain the program execution logic used in conjunction with the XR application and Monado runtime. Therefore, the technical solution of this invention has high adaptability and high scalability. Further combining this with the cross-platform nature of the WebRTC protocol allows this invention to meet modular expansion requirements.

[0103] In one embodiment, step S100 is included before step S120.

[0104] Step S100: Obtain the current application.

[0105] By obtaining the current application, the corresponding data to be rendered can be obtained according to the type of the current application, and used to determine the processing method of the data to be rendered (i.e., the target processing method).

[0106] For example, when the current application is a navigation application, the data to be rendered may include, but is not limited to: acceleration, angular velocity, scene image, scene depth image, position data, and radar data.

[0107] For example, when the current application is a translation application, the data to be rendered can be image data and / or voice data.

[0108] At this point, step S120 includes steps S121 and S122.

[0109] Step S121: Determine the target processing method based on the current application;

[0110] Step S122: Using the native Monado runtime, perform pre-rendering processing on the data to be rendered according to the target processing method.

[0111] Then, the processing method of the data to be rendered is determined according to the current application and recorded as the target processing method. Then, the native Monado runtime performs pre-rendering processing on the data to be rendered according to the target processing method.

[0112] For example, when the current application is a navigation application, the target determination processing method includes data fusion and spatial mapping processing. Specifically, through the native Monado runtime, acceleration, angular velocity, scene image, position data and radar data are fused to obtain the 6DoF (six degrees of freedom) pose of the XR device; at the same time, the scene depth image is spatially mapped to generate an environmental point cloud map and establish a virtual coordinate system.

[0113] For example, when the current application is a translation application, if the data to be rendered is voice data and the target processing method is determined to be voice recognition processing, then voice recognition processing is performed on the voice data to obtain semantic recognition results.

[0114] In this embodiment, by determining the processing method of the data to be rendered based on the current application, pre-processing of the data to be rendered can be performed, which facilitates further rendering processing.

[0115] Figure 4 This is the second flowchart of the rendering method provided by the present invention, as shown below. Figure 4 As shown, the rendering method is applied to a computing device, and the rendering method includes steps S210, S220 and S230.

[0116] Step S210: Receive rendering scene data sent by the XR device through the Monado streaming runtime, and submit the rendering scene data to the native XR application.

[0117] In this embodiment, the rendering method is applied to a computing device. The computing device can be a server, cloud server, PC, smartphone, tablet, or other similar device.

[0118] The computing device receives rendering scene data sent by the XR device through the streaming Monado runtime and submits the rendering scene data to the native XR application.

[0119] The Monado runtime is an open-source OpenXR runtime designed to provide immersive experiences. The Monado runtime refers to using Monado as the runtime environment for XR devices. Streaming, on the other hand, is a technology that transmits digital audio and video content in real time over the internet, allowing users to play it simultaneously during download without waiting for the entire file to finish downloading. A streaming Monado runtime refers to a modified or extended version of the open-source Monado runtime, enabling it to transmit the rendering results of XR applications running on computing devices to XR devices in real time, and to receive data to be rendered from XR devices to computing devices.

[0120] The rendered scene data is obtained by the XR device through the native Monado runtime after performing pre-rendering processing on the data to be rendered. The rendered scene data is sent by the XR device through the streaming XR application.

[0121] The data to be rendered can be acquired through sensors in the XR device, including but not limited to: attitude sensors, image sensors, GPS (Global Positioning System) sensors, LiDAR, and microphones. Attitude sensors include, but are not limited to, IMU (Inertial Measurement Unit), gyroscopes, accelerometers, and magnetometers. Image sensors include, but are not limited to, cameras, depth cameras, and eye-tracking sensors. The data to be rendered can also be obtained through applications on the XR device or computing device, including push notification applications such as calendars, schedule management apps, and health management apps. Correspondingly, the data to be rendered includes, but is not limited to: acceleration, angular velocity, magnetic field strength, magnetic field direction, scene images, scene depth images, location data, radar data, voice data, and push notification information.

[0122] As one implementation, the streaming Monado runtime can process the received rendering scene data into Monado's internal storage format, allowing native XR applications to obtain this data through the standard OpenXR interface.

[0123] Step S220: The rendering scene data is rendered using the native XR application to obtain a rendering result, and the rendering result is submitted to the streaming Monado runtime.

[0124] The rendering results are obtained by rendering scene data through native XR applications.

[0125] XR applications refer to various practical applications that utilize XR technology, such as navigation and translation applications that use augmented reality. Here, native XR applications refer to the various practical applications that originally used XR technology.

[0126] As one implementation method, the current application can be obtained, the data update scenario can be determined based on the current application, and then the rendering scenario data can be rendered by the native XR application according to the data update scenario to obtain the rendering result.

[0127] As one implementation method, after rendering is complete, the rendering result is returned to the streaming Monado runtime via the standard OpenXR interface.

[0128] Step S230: The rendering result is sent to the XR device via the Monado streaming runtime, so that the XR device displays the rendering result.

[0129] The rendering results are sent to the XR device via a streaming Monado runtime, specifically to the streaming XR application on the XR device. The XR device then receives the rendering results through the streaming XR application and submits them to the native Monado runtime for display.

[0130] The rendering method provided in this invention involves a computing device receiving rendering scene data sent by an XR device via a streaming Monado runtime, and submitting the rendering scene data to a native XR application. Then, the native XR application processes the rendering scene data to obtain a rendering result, which is then submitted to the streaming Monado runtime. Finally, the streaming Monado runtime sends the rendering result to the XR device for display. This invention, through secondary development of the Monado open-source framework, implements a cross-platform OpenXR runtime framework with remote rendering capabilities. Specifically, without modifying the native XR application and the native Monado runtime, it introduces additional customized streaming XR applications and streaming Monado runtimes, separating the native XR application to run on the computing device and the native Monado runtime to run on the XR device. In other words, the computing device includes a streaming Monado runtime and an integrated native XR application. The computing device receives rendering scene data from the XR device via the streaming Monado runtime, then processes the rendering scene data using the native XR application, and finally sends the processed rendering result back to the XR device via the streaming Monado runtime for display. By employing a high-performance computing device for rendering, this approach achieves rendering quality unattainable by traditional all-in-one XR devices, significantly enhancing the user's immersive experience. Furthermore, by offloading the high-power rendering task to the computing device and separating the rendering hardware from the XR device, the XR device can achieve longer battery life with a smaller battery capacity. It also effectively reduces the size and heat generation of the XR device, making it more comfortable for users to wear.

[0131] In one embodiment, the XR device includes a native Monado runtime and a streaming XR application; wherein, the native Monado runtime is used to perform pre-rendering processing on the data to be rendered to obtain rendering scene data; the streaming XR application is used to send the rendering scene data to the computing device, and is also used to receive the rendering results sent by the computing device and submit them to the native Monado runtime.

[0132] In this embodiment, the native Monado runtime refers to the Monado runtime originally used in the XR device. Streaming XR applications refer to an application mode that uses streaming technology to transmit XR content (such as rendering scene data, rendering results, etc.) between the XR device and the computing device, so that the rendering task is executed on the computing device, and then the rendering results are displayed and interacted with on the XR device.

[0133] XR devices use the native Monado runtime to perform pre-rendering processing on the data to be rendered, obtaining rendering scene data. Then, the rendering scene data is sent to the computing device via a streaming XR application, enabling the computing device to perform rendering processing. Subsequently, the streaming XR application receives the rendering results sent by the computing device and submits them to the native Monado runtime for display.

[0134] In this embodiment, without modifying the native XR application and native Monado runtime, additional customized streaming XR applications and streaming Monado runtimes are introduced. The native XR application runs on the computing device, and the native Monado runtime runs on the XR device. Correspondingly, the streaming XR application is introduced into the XR device. That is, the XR device includes both the native Monado runtime and the streaming XR application. The XR device only needs to perform pre-rendering processing, then send the processed rendering scene data to the computing device for rendering, and finally receive and display the rendering result returned by the computing device. Through this method, the XR device does not need to perform rendering processing itself; instead, it uses a high-performance computing device for rendering, achieving rendering quality unattainable by traditional all-in-one XR devices, thus significantly improving the user's immersive experience. Simultaneously, it allows the XR device to achieve longer battery life with a small-capacity battery, and effectively reduces the size and heat generation of the XR device, making it more comfortable for users to wear.

[0135] In one embodiment, step S220 includes:

[0136] The rendering results are sent to the XR device via the WebRTC protocol during the Monado streaming runtime.

[0137] Here, WebRTC is a real-time communication technology that allows web applications or websites to establish point-to-point connections between XR devices and computing devices without the need for an intermediary, enabling the transmission of data to be rendered or other arbitrary data. WebRTC can run in environments such as Web, Android, iOS, Windows, macOS, and Linux, thus allowing XR devices to connect to different types of computing units, such as PCs, cloud servers, and smartphones.

[0138] In this embodiment, by using the WebRTC protocol as the transmission protocol for the rendered results, low-latency performance requirements can be achieved. This significantly improves the user experience, especially for real-time scenarios.

[0139] Furthermore, as one implementation method, the streaming Monado runtime can overload Monado's original rendering and compositing process to extract the rendered content to be displayed on the XR device screen, encode the video using WebRTC (the encoding format can be H.264, H.265, VP8, or VP9), and then send it to the streaming XR application running on the XR device. Correspondingly, when the streaming XR application on the XR device receives the encoded rendering result, it decodes it and then submits the rendering result to the native Monado runtime through the standard OpenXR interface.

[0140] In this embodiment, by using the WebRTC protocol as the transmission protocol for the rendering results and by using mainstream video encoding formats such as H264 / H265 / VP8 / VP9 for video encoding of the rendering results, the data compression rate and rendering quality can be further guaranteed.

[0141] In one embodiment, step S220 includes:

[0142] Step S221: Obtain the current application and determine the data update scenario based on the current application;

[0143] Step S222: Using the native XR application, the rendering scene data is rendered based on the data update scene to obtain the rendering result.

[0144] In this embodiment, the specific process of obtaining the rendering result is as follows: First, the current application is obtained, and the data update scenario is determined based on the current application; then, the rendering scenario data is rendered by the native XR application according to the data update scenario to obtain the rendering result.

[0145] Among them, the data update scenario refers to the scenario that needs to be presented at the moment.

[0146] For example, when the current application is a navigation application, the rendered scene data includes 6DoF pose and environmental point cloud map, and the data update scene is determined to be the navigation scene that needs to be presented. Then, through the native XR application, the corresponding position is matched in the environmental point cloud map according to the 6DoF pose and position data, and the path planning is calculated at the same time, and 3D navigation arrows / path lines are superimposed on the real-world image of the camera.

[0147] For example, when the current application is a translation application, the rendered scene data is the speech recognition result, and the data update scene is determined to be the translation scene of the recognized target object. Then, through a native XR application, the speech recognition result is translated to obtain the translation result, and a floating text box is rendered on the surface of the recognized object.

[0148] In this embodiment, the data update scenario is determined based on the current application scenario to generate the corresponding rendering result, which facilitates subsequent display on XR devices.

[0149] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the above-described rendering method.

[0150] The electronic device can be an XR device or a computing device.

[0151] When the electronic device is an XR device, the rendering method includes: receiving data to be rendered acquired by the XR device; performing pre-rendering processing on the data to be rendered using the native Monado runtime, and submitting the obtained rendering scene data to a streaming XR application; sending the rendering scene data to a computing device through the streaming XR application, causing the computing device to perform rendering processing, and sending the obtained rendering result back to the XR device; receiving the rendering result sent by the computing device through the streaming XR application, and submitting the rendering result to the native Monado runtime; and displaying the rendering result through the native Monado runtime.

[0152] When the electronic device is a computing device, the rendering method includes: receiving rendering scene data sent by an XR device through a Monado streaming runtime, and submitting the rendering scene data to a native XR application; performing rendering processing on the rendering scene data through the native XR application to obtain a rendering result, and submitting the rendering result to the Monado streaming runtime; and sending the rendering result to the XR device through the Monado streaming runtime, so that the XR device displays the rendering result.

[0153] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. With this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the rendering methods provided by the above methods.

[0155] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the rendering methods provided by the methods described above.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A rendering method, characterized in that, Applied to XR devices, including native Monado runtime and streaming XR applications, the rendering method includes: Receive the data to be rendered acquired by the XR device; The native Monado runtime performs pre-rendering processing on the data to be rendered and submits the resulting rendering scene data to the streaming XR application. The streaming XR application sends the rendering scene data to the computing device, which then performs rendering processing and sends the resulting rendering result to the XR device. The streaming XR application receives the rendering results sent by the computing device and submits the rendering results to the native Monado runtime. The rendering results are displayed using the native Monado runtime.

2. The rendering method according to claim 1, characterized in that, The computing device includes a streaming Monado runtime and a native XR application; wherein, the streaming Monado runtime is used to receive rendering scene data sent by the streaming XR application; The native XR application is used to render the scene data to obtain the rendering result.

3. The rendering method according to claim 1, characterized in that, Sending the rendered scene data to the computing device via the streaming XR application includes: The streaming XR application sends the rendered scene data to the computing device via the WebRTC protocol.

4. The rendering method according to claim 1, characterized in that, Before performing pre-rendering processing on the data to be rendered using the native Monado runtime, the process also includes: Get the current application; The pre-rendering processing of the data to be rendered using the native Monado runtime includes: Based on the current application, determine the target processing method; The native Monado runtime performs pre-rendering processing on the data to be rendered according to the target processing method.

5. A rendering method, characterized in that, Applied to computing devices, including streaming Monado runtimes and native XR applications, the rendering method includes: The Monado streaming runtime receives rendering scene data sent by the XR device and submits the rendering scene data to the native XR application. The rendering scene data is rendered using the native XR application to obtain the rendering result, and the rendering result is submitted to the streaming Monado runtime. The rendering result is sent to the XR device via the Monado streaming runtime, so that the XR device can display the rendering result.

6. The rendering method according to claim 5, characterized in that, The XR device includes a native Monado runtime and a streaming XR application; wherein, the native Monado runtime is used to perform pre-rendering processing on the data to be rendered to obtain rendering scene data; The streaming XR application is used to send the rendered scene data to the computing device, and also to receive the rendering results sent by the computing device and submit them to the native Monado runtime.

7. The rendering method according to claim 5, characterized in that, Sending the rendering result to the XR device via the Monado streaming runtime includes: The rendering results are sent to the XR device via the WebRTC protocol during the Monado streaming runtime.

8. The rendering method according to claim 5, characterized in that, The rendering process of the rendering scene data using a native XR application to obtain the rendering result includes: Obtain the current application and determine the data update scenario based on the current application; The native XR application renders the scene data based on the updated scene data to obtain the rendering result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the rendering method as described in any one of claims 1 to 8.

10. A rendering system, characterized in that, It includes XR devices and computing devices, wherein the XR devices include a native Monado runtime and streaming XR applications, and the computing devices include a streaming Monado runtime and native XR applications, wherein: The XR device is used to receive the data to be rendered acquired by the XR device, perform pre-rendering processing on the data to be rendered through the native Monado runtime to obtain rendering scene data, and send the rendering scene data to the computing device through the streaming XR application. The computing device is configured to receive the rendering scene data through the streaming Monado runtime, perform rendering processing on the rendering scene data through the native XR application to obtain a rendering result, and send the rendering result to the XR device through the streaming Monado runtime. The XR device is used to receive the rendering results through the streaming XR application and display the rendering results through the native Monado runtime.

Images