Rendering method, electronic equipment and system

Abstract

The invention provides a rendering method, an electronic device and a system, and relates to the technical field of rendering, the method is applied to an XR device, and the method comprises the following steps: receiving to-be-rendered data acquired by the XR device; performing rendering preprocessing on the to-be-rendered data during operation of the native monado, and submitting obtained rendering scene data to a streaming XR application; sending the rendering scene data to a computing device through the streaming XR application, enabling the computing device to carry out rendering processing, and sending an obtained rendering result to the XR device; a rendering result sent by the computing device is received through the streaming XR application, and the rendering result is submitted to the native monado when the native monado runs; and the rendering result is displayed when the native monado runs. The problems that an existing XR head-mounted display is poor in performance, short in endurance and poor in wearing comfort can be solved.

Description

Technical Field

[0001] The present invention relates to the field of rendering technology, and in particular to a rendering method, electronic equipment, and system. Background Art

[0002] Extended Reality (XR) technology, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), relies on high-performance, low-latency, real-time 3D rendering capabilities for its core experience, creating a realistic and immersive environment. Currently, mainstream consumer XR devices generally adopt a standalone head-mounted display (HMD) design, which integrates the computing unit, display unit, sensors, and battery into the headset worn by the user. While this design improves portability and deployment ease, it introduces significant and irreconcilable contradictions in rendering performance, battery life, and thermal management, becoming a key bottleneck restricting the upgrade of the XR experience: (1) The contradiction between performance and portability: Due to the internal space constraints and heat dissipation requirements of the integrated head-mounted device, it is impossible to use a high-performance core processor, resulting in an insurmountable technical gap between rendering capabilities and portability; (2) Poor battery life: In order to make users wear it comfortably, the all-in-one XR headset has strict weight restrictions, so it is impossible to use a large-capacity battery, which greatly limits the battery life of XR devices; (3) Heat dissipation bottleneck: High-performance rendering generates a lot of heat. In a highly integrated, space-enclosed, and all-in-one XR headset that fits closely to the user's face, the user will feel uncomfortable when wearing it.

[0003] Therefore, how to solve the core contradictions of XR headsets in terms of performance, battery life and wearing comfort at the current stage is an urgent problem that needs to be solved. Summary of the Invention

[0004] The present invention provides a rendering method, electronic device, and system to solve the problems of poor performance, short battery life, and poor wearing comfort of existing XR head displays.

[0005] The present invention provides a rendering method, applied to an XR device, comprising: Receiving data to be rendered obtained by the XR device; Performing pre-rendering processing on the data to be rendered through the native monado runtime, and submitting the resulting rendered scene data to the streaming XR application; Sending the rendered scene data to a computing device through the streaming XR application, causing the computing device to perform rendering processing, and sending the obtained rendering result to the XR device; receiving, through the streaming XR application, a rendering result sent by the computing device, and submitting the rendering result to the native monado runtime; The rendered result is displayed by the native monado runtime.

[0006] According to a rendering method provided by the present invention, the computing device includes a streaming monado runtime and a native XR application; wherein the streaming monado runtime is used to receive rendering scene data sent by the streaming XR application; The native XR application is used to render the rendering scene data to obtain a rendering result.

[0007] According to a rendering method provided by the present invention, sending the rendered scene data to a computing device through the streaming XR application includes: Through the streaming XR application, the rendered scene data is sent to the computing device via the web real-time communication WebRTC protocol.

[0008] According to a rendering method provided by the present invention, before performing pre-rendering processing on the data to be rendered by the native Monado runtime, the method further includes: Get the current application; The pre-rendering processing of the data to be rendered by the native monado runtime includes: Determining a target processing method based on the current application; When the native monado is running, the data to be rendered is pre-processed according to the target processing mode.

[0009] The present invention provides a rendering method, applied to a computing device, comprising: Receive rendered scene data sent by the XR device through the streaming monado runtime, and submit the rendered scene data to the native XR application; Rendering the rendering scene data through the native XR application to obtain a rendering result, and submitting the rendering result to the streaming monado runtime; The rendering result is sent to the XR device through the streaming monado runtime, so that the XR device displays the rendering result.

[0010] According to a rendering method provided by the present invention, the XR device includes a native monado runtime and a streaming XR application; wherein the native monado runtime is used to perform pre-rendering processing on data to be rendered to obtain rendering scene data; The streaming XR application is used to send the rendered scene data to the computing device, and is also used to receive the rendering results sent by the computing device and submit them to the native monado runtime.

[0011] According to a rendering method provided by the present invention, sending the rendering result to the XR device through the streaming monado runtime includes: When the streaming monado is running, the rendering result is sent to the XR device through the WebRTC protocol.

[0012] According to a rendering method provided by the present invention, the rendering scene data is rendered by a native XR application to obtain a rendering result, including: Obtaining a current application, and determining a data update scenario based on the current application; The native XR application is used to render the rendering scene data according to the data update scene to obtain a rendering result.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above rendering methods when executing the computer program.

[0014] The present invention also provides a rendering system, comprising an XR device and a computing device, wherein: The XR device is configured to receive the to-be-rendered data acquired by the XR device, perform pre-rendering processing on the to-be-rendered data through a native monado runtime to obtain rendering scene data, and send the rendering scene data to a computing device through a streaming XR application; The computing device is configured to receive the rendering scene data through a streaming monado runtime, render the rendering scene data through a native XR application to obtain a rendering result, and send the rendering result to the XR device through the streaming monado runtime; The XR device is configured to receive the rendering result through the streaming XR application and display the rendering result through the native monado runtime.

[0015] The rendering method, electronic device, and system provided by the present invention obtain data to be rendered through an XR device, perform rendering pre-processing on the data to be rendered through the native monado runtime, and submit the obtained rendering scene data to a streaming XR application; send the rendering scene data to a computing device through the streaming XR application, enable the computing device to perform rendering processing, and send the obtained rendering result to the XR device; then, receive the rendering result sent by the computing device through the streaming XR application, and submit the rendering result to the native monado runtime; finally, display the rendering result through the native monado runtime. The present invention implements a cross-platform OpenXR runtime framework with remote rendering capabilities through secondary development of the monado open source framework. Specifically, without changing the native XR application and the native monado runtime, by introducing additional customized streaming XR applications and streaming monado runtime, the native XR application is separated to run on the computing device, and the native monado runtime is separated to run on the XR device. That is, the XR device includes the native monado runtime and the introduced streaming XR application. The XR device performs rendering pre-processing through the native monado runtime, and sends the processed rendering scene data to the computing device through the streaming XR application, and the computing device performs rendering processing. In the above way, by using high-performance computing devices for rendering processing, it is possible to obtain rendering quality that cannot be obtained by traditional all-in-one XR devices, thereby significantly improving the user's immersive experience. At the same time, by transferring high-power rendering tasks to the computing device and separating the rendering hardware from the XR device, the XR device can achieve higher battery life with a small-capacity battery, and can also effectively reduce the size and heat generation of the XR device, making it more comfortable for users to wear. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a standalone rendering framework diagram for existing XR devices; Figure 2 is a system architecture diagram of the rendering system provided by the present invention; Figure 3 This is one of the flowcharts of the rendering method provided by the present invention; Figure 4 This is the second flowchart of the rendering method provided by the present invention; Figure 5It is a structural schematic diagram of the electronic device provided by the present invention; Wherein, the accompanying drawings are marked as follows: 01: Native XR application; 02: Native monado runtime; 03: Streaming XR application; 04: Streaming monado runtime. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0019] Extended Reality (XR) technology, including virtual reality (VR), augmented reality (AR), and mixed reality (MR), relies on high-performance, low-latency, real-time 3D rendering capabilities for its core experience, creating a realistic and immersive environment. Currently, mainstream consumer XR devices generally adopt a standalone head-mounted display (HMD) design, which integrates the computing unit, display unit, sensors, and battery into the headset worn by the user. While this design improves portability and deployment ease, it introduces significant and irreconcilable contradictions in rendering performance, battery life, and thermal management, becoming a key bottleneck restricting the upgrade of the XR experience: (1) The contradiction between performance and portability: Due to the internal space constraints and heat dissipation requirements of the integrated head-mounted device, it is impossible to use a high-performance core processor, resulting in an insurmountable technical gap between rendering capabilities and portability; (2) Poor battery life: In order to make users wear it comfortably, the all-in-one XR headset has strict weight restrictions, so it is impossible to use a large-capacity battery, which greatly limits the battery life of XR devices; (3) Heat dissipation bottleneck: High-performance rendering generates a lot of heat. In a highly integrated, space-enclosed, and all-in-one XR headset that fits closely to the user's face, the user will feel uncomfortable when wearing it.

[0020] Therefore, how to solve the core contradictions of XR headsets in terms of performance, battery life and wearing comfort at the current stage is an urgent problem that needs to be solved.

[0021] To address these issues, existing technologies have proposed solutions using external computing units, such as the wireless streaming feature of the HTC Vive (a VR headset jointly developed by HTC and Valve). However, this solution essentially ties the all-in-one XR headset to a fixed external computing unit (usually the user's PC), preventing easy replacement with a computing unit with different features (such as a portable smartphone) based on user needs. Therefore, this solution only partially alleviates the computing power issue and does not truly address the need for modular expansion.

[0022] Based on the above problems, the present invention proposes a rendering method, electronic device and system. Figure 1-Figure 5 Provide a description.

[0023] Figure 1 It is a standalone rendering framework diagram of existing XR devices. Figure 1 As shown, the existing XR device adopts an independent XR rendering framework. After obtaining the data to be rendered, the XR device performs pre-rendering processing on the data to be rendered through the native monado runtime 02, and then sends the processed rendering scene data to the native XR application 01. The native XR application 01 renders the rendering scene data to obtain the rendering result, and the rendering result is displayed through the native monado runtime 02.

[0024] Based on the above Figure 1 , the present invention, without changing the native XR application and native monado runtime, introduces additional customized streaming XR application 03 and streaming monado runtime 04, and separates the native XR application 01 and native monado runtime 02 to run on the computing device and XR device respectively. Figure 2 , Figure 2 This is a system architecture diagram of the rendering system provided by the present invention. Figure 2 As shown, the rendering system includes an extended reality (XR) device 10 and a computing device 20 .

[0025] The XR device 10 includes, but is not limited to, a VR (Virtual Reality) device, an AR (Augmented Reality) device, and an MR (Mixed Reality) device, and is used to perform the rendering method of the present invention. The computing device 20 can be a server, a cloud server, a PC (Personal Computer), a smartphone, a tablet computer, or other device.

[0026] In this rendering system: The XR device 10 is configured to receive data to be rendered acquired by the XR device, perform pre-rendering processing on the data to be rendered using a native monado runtime to obtain rendering scene data, and send the rendering scene data to a computing device via a streaming XR application; The computing device 20 is configured to receive the rendering scene data through a streaming monado runtime, render the rendering scene data through a native XR application to obtain a rendering result, and send the rendering result to the XR device through the streaming monado runtime; The XR device 10 is configured to receive the rendering result through the streaming XR application and display the rendering result through the native monado runtime.

[0027] In this embodiment, the XR device obtains the data to be rendered, performs pre-rendering processing on the data to be rendered through the native monado runtime, obtains the rendered scene data, and submits it to the streaming XR application. Then, the rendered scene data is sent to the computing device through the streaming XR application. The data to be rendered includes but is not limited to: acceleration, angular velocity, magnetic field strength, magnetic field direction, scene image, scene depth image, position data, radar data, voice data and push information. The native monado runtime refers to the monado runtime originally used in the XR device. Streaming XR application refers to an application mode that transmits XR content (such as rendering scene data, rendering results, etc.) between the XR device and the computing device through streaming technology, so as to put the rendering task on the computing device for execution and then display and interact with the rendering results on the XR device.

[0028] After the computing device obtains the rendered scene data through the streaming monado runtime, it submits it to the native XR application. The native XR application then renders the rendered scene data to obtain the rendered result, which is then submitted to the streaming monado runtime. The streaming monado runtime refers to the modification or extension of the open source monado runtime, which enables it to transmit the rendering results of the XR application running on the computing device to the XR device in real time and receive the data to be rendered transmitted by the XR device to the computing device. Native XR applications refer to various practical applications that originally used XR technology.

[0029] The compute device then sends the rendered results to the XR device via the streaming monado runtime.

[0030] Finally, the XR device receives the rendering results through the streaming XR application and submits them to the native monado runtime, which then displays the rendering results through the native monado runtime.

[0031] It should be understood that the rendering system described in the present invention and the rendering method described below correspond to each other. That is, the specific processing process of the XR device and the computing device can refer to the following embodiments and will not be repeated here.

[0032] The present invention provides a rendering system, which includes an XR device and a computing device. Compared with the traditional XR device that adopts the native monado runtime and the independent rendering framework of the native XR application, the present invention realizes a cross-platform OpenXR runtime framework with remote rendering capabilities by secondary development of the monado open source framework. Specifically, without changing the native XR application and the native monado runtime, by introducing additional customized streaming XR applications and streaming monado runtime, the native XR application is separated to the computing device for operation, and the native monado runtime is separated to the XR device for operation. Correspondingly, the streaming XR application is introduced into the XR device, and the streaming monado runtime is introduced into the computing device. The XR device performs rendering pre-processing through the native monado runtime, and sends the processed rendering scene data to the computing device through the streaming XR application, and the computing device performs rendering processing. In the above manner, a high-performance computing device is used for rendering processing, and a rendering quality that cannot be obtained by traditional all-in-one XR devices can be obtained, thereby significantly improving the user's immersive experience. At the same time, by transferring high-power rendering tasks to computing devices and separating the rendering hardware from XR devices, XR devices can achieve longer battery life with small-capacity batteries. It can also effectively reduce the size and heat generation of XR devices, making users more comfortable when wearing them.

[0033] Furthermore, because the primary function of the streaming XR application and the streaming monado runtime in this invention is to transmit rendered scene data and rendering results, the actual XR experience still comes from the native XR application and the native monado runtime. In other words, the present invention retains the program execution logic of the XR application and the monado runtime in both computing devices and XR devices, thus making the technical solution of this invention highly adaptable and scalable.

[0034] Figure 3 This is one of the flowcharts of the rendering method provided by the present invention, such as Figure 3 As shown, the rendering method includes: step S110, step S120, step S130, step S140 and step S150.

[0035] Step S110: receiving the data to be rendered obtained by the XR device.

[0036] In this embodiment, the rendering method is applied to an XR device.

[0037] XR devices refer to wearable or portable devices that achieve human-computer interaction by integrating virtual and real environments through hardware and software technologies. XR devices include but are not limited to: VR devices, AR devices, and MR devices. Among them, VR devices use computer technology to simulate and generate three-dimensional virtual spaces, allowing users to immerse themselves in and interact with them for an immersive experience; AR devices use technology to integrate virtual information with the real world, superimposing it on real scenes in real time to enhance the sensory experience; MR devices mix the real world and the virtual world to produce a new visual environment that contains both physical entities and virtual information, and users can interact with these physical entities and virtual information in real time. The XR device can be a head-mounted XR device, such as AR glasses, VR glasses, etc.

[0038] Here, the data to be rendered can be collected through the sensors of the XR device, including but not limited to: attitude sensors, image sensors, GPS (Global Positioning System, GPS) sensors, lidars and microphones, among which attitude sensors include but not limited to IMU (Inertial Measurement Unit), gyroscopes, accelerometers and magnetometers, and image sensors include but not limited to cameras, depth cameras and eye tracking sensors. The data to be rendered can also be obtained through applications of the XR device or computing device, and the applications include push-type programs, such as calendars, schedule management applications, and health management applications. Correspondingly, the data to be rendered includes but is not limited to: acceleration, angular velocity, magnetic field strength, magnetic field direction, scene image, scene depth image, location data, radar data, voice data and push information.

[0039] As an implementation, the current application may be first acquired, and then, based on the type of the current application, the corresponding data to be rendered may be acquired. For example, if the current application is a navigation application, the acquired data to be rendered may include, but is not limited to, acceleration, angular velocity, scene image, scene depth image, location data, and radar data. For another example, if the current application is a translation application, the acquired data to be rendered may be voice data and / or scene image.

[0040] Step S120 : Pre-rendering processing is performed on the data to be rendered through the native monado runtime, and the obtained rendering scene data is submitted to the streaming XR application.

[0041] The XR device includes a native monado runtime and streaming XR applications.

[0042] The Monado runtime is an open-source OpenXR runtime designed to provide immersive experiences. The Monado runtime refers to the runtime environment for XR devices using Monado. Here, the native Monado runtime refers to the Monado runtime natively used by XR devices.

[0043] Streaming is a technology for transmitting digital audio and video content in real time over the internet, allowing users to play content simultaneously while it's being downloaded, without having to wait for the entire file to download. XR applications refer to various practical applications that utilize XR technology, such as navigation and translation apps displayed through augmented reality. Streaming XR applications use streaming technology to transmit XR content (such as rendered scene data and rendering results) between XR devices and computing devices, offloading rendering tasks to the computing device and then displaying and interacting with the rendered results on the XR device.

[0044] The native Monado runtime performs pre-rendering processing on the data to be rendered to obtain the rendered scene data. The specific processing method can be determined according to the current application.

[0045] Specifically, the current application can be retrieved, a target processing method can be determined based on the current application, and then the native Monado runtime can perform pre-rendering processing on the data to be rendered based on the target processing method. For example, if the current application is a translation application and the corresponding data to be rendered is voice data, the target processing method can be determined to be voice recognition processing. In this case, the native Monado runtime will perform voice recognition processing on the data to be rendered to obtain the voice recognition results.

[0046] After obtaining the rendered scene data, the rendered scene data is submitted to the streaming XR application.

[0047] As an implementation, streaming XR applications can obtain these rendered scene data from the native monado runtime through the standard OpenXR (a programming interface for XR applications) interface.

[0048] Step S130: Send the rendered scene data to a computing device through the streaming XR application, so that the computing device performs rendering processing and sends the obtained rendering result to the XR device.

[0049] The rendered scene data is sent to the computing device through the streaming XR application. Correspondingly, after receiving the rendered scene data, the computing device renders the rendered scene data to obtain a rendering result, and then sends the rendering result to the XR device.

[0050] As an implementation, a streaming XR application may send rendered scene data to a computing device via wired transmission.

[0051] As another implementation, a streaming XR application may send rendered scene data to a computing device via Wi-Fi (Wireless Fidelity) communication.

[0052] As another implementation, streaming XR applications can send rendered scene data to a computing device via the WebRTC (Web Real-Time Communications) protocol. WebRTC is a real-time communication technology that allows web applications or websites to establish a peer-to-peer connection between XR devices and computing devices without the need for an intermediary, enabling the transmission of rendered scene data, rendering results, or any other arbitrary data.

[0053] Step S140 : receiving the rendering result sent by the computing device through the streaming XR application, and submitting the rendering result to the native monado runtime.

[0054] The XR device can receive the rendering results sent by the computing device through the streaming XR application, and then submit the rendering results to the native monado runtime.

[0055] Specifically, the rendering results can be submitted to the native monado runtime through the standard OpenXR interface.

[0056] Step S150: displaying the rendering result through the native monado runtime.

[0057] After the native monado runtime receives the rendering results, it displays the rendering results through the native monado runtime.

[0058] The rendering method provided by the embodiment of the present invention obtains data to be rendered through an XR device, performs rendering pre-processing on the data to be rendered through the native monado runtime, and submits the obtained rendering scene data to a streaming XR application; sends the rendering scene data to a computing device through the streaming XR application, causes the computing device to perform rendering processing, and sends the obtained rendering result to the XR device; then, receives the rendering result sent by the computing device through the streaming XR application, and submits the rendering result to the native monado runtime; finally, displays the rendering result through the native monado runtime. The present invention implements a cross-platform OpenXR runtime framework with remote rendering capabilities through secondary development of the monado open source framework. Specifically, without changing the native XR application and the native monado runtime, by introducing additional customized streaming XR applications and streaming monado runtime, the native XR application is separated to run on the computing device, and the native monado runtime is separated to run on the XR device. That is, the XR device includes the native monado runtime and the introduced streaming XR application. The XR device performs rendering pre-processing through the native monado runtime, and sends the processed rendering scene data to the computing device through the streaming XR application, and the computing device performs rendering processing. In the above way, by using high-performance computing devices for rendering processing, it is possible to obtain rendering quality that cannot be obtained by traditional all-in-one XR devices, thereby significantly improving the user's immersive experience. At the same time, by transferring high-power rendering tasks to the computing device and separating the rendering hardware from the XR device, the XR device can achieve higher battery life with a small-capacity battery, and can also effectively reduce the size and heat generation of the XR device, making it more comfortable for users to wear.

[0059] In one embodiment, the computing device includes a streaming monado runtime and a native XR application; wherein the streaming monado runtime is used to receive rendering scene data sent by the streaming XR application; and the native XR application is used to render the rendering scene data to obtain a rendering result.

[0060] In this embodiment, the streaming Monado runtime refers to the modification or extension of the open-source Monado runtime to enable it to transmit the rendering results of XR applications running on a computing device to the XR device in real time, and receive the rendering data transmitted by the XR device to the computing device. Native XR applications refer to various practical applications that originally used XR technology.

[0061] After receiving the rendered scene data from the streaming XR application via the streaming monado runtime, the computing device uses the native XR application to obtain the rendered scene data from the streaming monado runtime. Specifically, the computing device can obtain the rendered scene data from the streaming monado runtime via the standard OpenXR interface. The computing device then renders the rendered scene data to obtain the rendered result, which is then returned to the streaming monado runtime of the XR device via the standard OpenXR interface.

[0062] In this embodiment, without changing the native XR application and the native monado runtime, by introducing an additional customized streaming XR application and streaming monado runtime, the native XR application is separated to the computing device for operation, and the native monado runtime is separated to the XR device for operation. Correspondingly, the streaming monado runtime is introduced to the computing device. That is, the computing device includes the streaming monado runtime and the native XR application. The computing device performs rendering processing and then sends it to the XR device. In the above manner, high-performance computing devices are used for rendering processing, which can obtain rendering quality that cannot be obtained by traditional all-in-one XR devices, thereby significantly improving the user's immersive experience. At the same time, the XR device can achieve higher battery life with a small-capacity battery, and can also effectively reduce the volume and heat generation of the XR device, making it more comfortable for users to wear.

[0063] In one embodiment, step S130 includes: Through the streaming XR application, the rendered scene data is sent to the computing device via the web real-time communication WebRTC protocol.

[0064] The WebRTC protocol is a real-time communication technology that allows web applications or websites to establish point-to-point connections between XR devices and computing devices without the need for an intermediary, enabling the transmission of rendered data or any other arbitrary data. WebRTC runs on the Web, Android, iOS, Windows, macOS, Linux, and other environments, allowing XR devices to connect to different types of computing devices, such as PCs, cloud servers, and smartphones.

[0065] In this embodiment, by adopting the WebRTC protocol as the transmission protocol for rendering scene data, the low-latency performance requirement can be achieved. Especially for real-time scenarios, the user experience can be significantly improved. In addition, since the main function of the streaming XR application and the streaming monado runtime of the present invention is transmission, both the computing device and the XR device retain the program operation logic used in conjunction with the XR application and the monado runtime. Therefore, the technical solution of the present invention has the characteristics of high adaptability and high scalability. Further combined with the cross-platform characteristics of the WebRTC protocol, the present invention can realize modular expansion requirements.

[0066] In one embodiment, before step S120, step S100 is further included.

[0067] Step S100: Get the current application.

[0068] By obtaining the current application, the corresponding data to be rendered can be obtained according to the type of the current application, and used to determine the processing method (ie, target processing method) of the data to be rendered.

[0069] Exemplarily, when the current application is a navigation application, the acquired data to be rendered may include, but is not limited to: acceleration, angular velocity, scene image, scene depth image, position data, and radar data.

[0070] Exemplarily, when the current application is a translation application, the acquired data to be rendered may be image data and / or voice data.

[0071] At this time, step S120 includes step S121 and step S122.

[0072] Step S121, determining a target processing method according to the current application; Step S122 : When the native monado is running, pre-rendering processing is performed on the data to be rendered according to the target processing mode.

[0073] Then, the processing method of the data to be rendered is determined according to the current application, recorded as the target processing method, and then the native monado runtime performs pre-rendering processing on the data to be rendered according to the target processing method.

[0074] For example, when the current application is a navigation application, the target processing method includes data fusion and spatial mapping processing. Specifically, through the native monado runtime, the acceleration, angular velocity, scene image, position data and radar data are fused to obtain the 6DoF (six degrees of freedom) posture of the XR device; at the same time, the scene depth image is spatially mapped to generate an environment point cloud map and establish a virtual coordinate system.

[0075] For example, when the current application is a translation application, if the data to be rendered is voice data and the target processing method is determined to be voice recognition processing, voice recognition processing is performed on the voice data to obtain a semantic recognition result.

[0076] In this embodiment, by determining the processing method of the data to be rendered according to the current application, the data to be rendered is pre-processed, which can facilitate further rendering processing later.

[0077] Figure 4 This is the second flow chart of the rendering method provided by the present invention. Figure 4 As shown, the rendering method is applied to a computing device, and the rendering method includes: step S210, step S220 and step S230.

[0078] Step S210 : Receive the rendering scene data sent by the XR device through the streaming monado runtime, and submit the rendering scene data to the native XR application.

[0079] In this embodiment, the rendering method is applied to a computing device, which may be a server, a cloud server, a PC, a smartphone, a tablet computer, or the like.

[0080] The computing device receives the rendered scene data sent by the XR device through the streaming monado runtime and submits the rendered scene data to the native XR application.

[0081] The monado runtime is an open-source OpenXR runtime designed to provide immersive experiences. The monado runtime refers to using monado as the runtime environment for XR devices. Streaming is a technology for transmitting digital audio and video content over the internet in real time, allowing users to play it while it's being downloaded, without having to wait for the entire file to download. The streaming monado runtime is a modification or extension of the open-source monado runtime, enabling it to transmit rendering results from XR applications running on a computing device to the XR device in real time, and to receive rendering data transmitted by the XR device to the computing device.

[0082] Rendering scene data is obtained by the XR device after pre-rendering processing of the rendering data through the native Monado runtime. The rendering scene data is sent by the XR device through the streaming XR application.

[0083] The data to be rendered can be collected through the sensors of the XR device, including but not limited to: attitude sensors, image sensors, GPS (Global Positioning System, GPS) sensors, lidars and microphones, among which attitude sensors include but not limited to IMU (Inertial Measurement Unit), gyroscopes, accelerometers and magnetometers, and image sensors include but not limited to cameras, depth cameras and eye tracking sensors. The data to be rendered can also be obtained through applications of the XR device or computing device, and the applications include push-type programs, such as calendars, schedule management applications, and health management applications. Correspondingly, the data to be rendered includes but is not limited to: acceleration, angular velocity, magnetic field strength, magnetic field direction, scene image, scene depth image, location data, radar data, voice data and push information.

[0084] As an implementation, the streaming monado runtime may process the received rendered scene data into monado's internal storage format, so that native XR applications can obtain this data through the standard OpenXR interface.

[0085] Step S220 : Rendering the rendering scene data through the native XR application to obtain a rendering result, and submitting the rendering result to the streaming monado runtime.

[0086] The rendering scene data is rendered and processed through the native XR application to obtain the rendering results.

[0087] XR applications refer to various practical applications that use XR technology, such as navigation applications and translation applications displayed through augmented reality. Here, native XR applications refer to various practical applications that originally use XR technology.

[0088] As an implementation method, the current application can be obtained, and the data update scene can be determined according to the current application. Then, the rendering scene data can be rendered according to the data update scene through the native XR application to obtain the rendering result.

[0089] As an implementation, after rendering is completed, the rendering result is returned to the streaming monado runtime through the standard OpenXR interface.

[0090] Step S230 : Sending the rendering result to the XR device through the streaming monado runtime, so that the XR device displays the rendering result.

[0091] The rendering results are sent to the XR device through the streaming monado runtime, specifically to the XR device's streaming XR application. The XR device receives the rendering results through the streaming XR application and submits them to the native monado runtime, which then displays them.

[0092] The rendering method provided by an embodiment of the present invention is that a computing device receives rendering scene data sent by an XR device through a streaming monado runtime, and submits the rendering scene data to a native XR application; then, the rendering scene data is rendered and processed by the native XR application to obtain a rendering result, and the rendering result is submitted to the streaming monado runtime; finally, the rendering result is sent to the XR device through the streaming monado runtime, so that the XR device can display the rendering result. The present invention implements a cross-platform OpenXR runtime framework with remote rendering capabilities through secondary development of the monado open source framework. Specifically, without changing the native XR application and the native monado runtime, by introducing additional customized streaming XR applications and streaming monado runtime, the native XR application is separated to run on the computing device, and the native monado runtime is separated to run on the XR device. That is, the computing device includes a streaming monado runtime and an introduced native XR application. The computing device receives the rendering scene data sent by the XR device through the streaming monado runtime, and then renders the rendering scene data through the native XR application, and then sends the processed rendering results to the XR device through the streaming monado runtime, and the XR device displays the rendering results. In the above manner, high-performance computing devices are used for rendering processing, which can achieve rendering quality that cannot be obtained by traditional all-in-one XR devices, thereby significantly improving the user's immersive experience. At the same time, by transferring high-power rendering tasks to the computing device and separating the rendering hardware from the XR device, the XR device can achieve higher battery life with a small-capacity battery, and can also effectively reduce the volume and heat generation of the XR device, making it more comfortable for users to wear.

[0093] In one embodiment, the XR device includes a native monado runtime and a streaming XR application; wherein the native monado runtime is used to perform pre-rendering processing on the data to be rendered to obtain rendering scene data; the streaming XR application is used to send the rendering scene data to the computing device, and is also used to receive the rendering results sent by the computing device and submit them to the native monado runtime.

[0094] In this embodiment, the native Monado runtime refers to the Monado runtime originally used by XR devices. Streaming XR applications use streaming technology to transmit XR content (such as rendered scene data and rendering results) between XR devices and computing devices, offloading rendering tasks to the computing device and then displaying and interacting with the rendering results on the XR device.

[0095] The XR device performs pre-rendering processing on the data to be rendered through the native monado runtime to obtain the rendered scene data; then, the rendered scene data is sent to the computing device through the streaming XR application, so that the computing device performs rendering processing; further, the rendering results sent by the computing device are received through the streaming XR application and submitted to the native monado runtime to display the rendering results through the native monado runtime.

[0096] In this embodiment, without changing the native XR application and the native monado runtime, by introducing an additional customized streaming XR application and streaming monado runtime, the native XR application is separated to the computing device for operation, and the native monado runtime is separated to the XR device for operation, and correspondingly, the streaming XR application is introduced to the XR device. That is, the XR device includes the native monado runtime and the streaming XR application. The XR device only needs to perform pre-rendering processing, and then send the processed rendering scene data to the computing device, which performs rendering processing, and then receives the rendering results returned by the computing device and displays them. In this way, the XR device does not need to perform rendering processing, but uses a high-performance computing device for rendering processing, which can obtain rendering quality that traditional all-in-one XR devices cannot obtain, thereby significantly improving the user's immersive experience. At the same time, the XR device can achieve higher battery life with a small-capacity battery, and can also effectively reduce the volume and heat generation of the XR device, making it more comfortable for the user to wear.

[0097] In one embodiment, step S220 includes: When the streaming monado is running, the rendering result is sent to the XR device through the WebRTC protocol.

[0098] The WebRTC protocol is a real-time communication technology that allows web applications or websites to establish point-to-point connections between XR devices and computing devices without the need for an intermediary, enabling the transmission of rendered data or any other arbitrary data. WebRTC runs on the Web, Android, iOS, Windows, macOS, Linux, and other environments, allowing XR devices to connect to different types of computing units, such as PCs, cloud servers, and smartphones.

[0099] In this embodiment, by adopting the WebRTC protocol as the transmission protocol for rendering results, low-latency performance requirements can be achieved, which can significantly improve the user experience, especially for real-time scenarios.

[0100] Furthermore, as an implementation, the streaming monado runtime can overload monado's existing rendering and composition process to intercept the rendered content to be displayed on the XR device screen, encode the video using WebRTC (in H264, H265, VP8, or VP9), and then send it to the streaming XR application running on the XR device. In turn, the streaming XR application on the XR device decodes the encoded rendering result upon receiving it, and then submits the rendering result to the native monado runtime via the standard OpenXR interface.

[0101] In this embodiment, by adopting the WebRTC protocol as the transmission protocol for the rendering results, and encoding the rendering results using mainstream video encoding formats such as H264 / H265 / VP8 / VP9, the data compression rate and rendering quality can be further guaranteed.

[0102] In one embodiment, step S220 includes: Step S221, obtaining the current application, and determining the data update scenario according to the current application; Step S222: Render the rendering scene data according to the data update scene through the native XR application to obtain a rendering result.

[0103] In this embodiment, the specific process of obtaining the rendering result is as follows: first obtain the current application, and determine the data update scene based on the current application; then, use the native XR application to render the rendering scene data according to the data update scene to obtain the rendering result.

[0104] The data update scenario refers to the scenario that needs to be presented currently.

[0105] For example, if the current application is a navigation application, the rendered scene data includes 6DoF poses and an environmental point cloud map. The data update scene is determined to be the navigation scene currently being rendered. Then, the native XR application matches the corresponding position in the environmental point cloud map based on the 6DoF pose and position data, calculates the path plan, and overlays 3D navigation arrows / path lines on the real-world camera image.

[0106] For example, if the current application is a translation application, the rendered scene data is the speech recognition result, and the data update scene is determined to be the translation scene of the recognized target object. Then, the native XR application translates the speech recognition result to obtain the translation result, and renders a floating text box on the surface of the recognized object.

[0107] In this embodiment, by determining the data update scenario according to the current application scenario to generate the corresponding rendering results, it can be facilitated for subsequent display on the XR device.

[0108] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the above-mentioned rendering method.

[0109] The electronic device can be an XR device or a computing device.

[0110] When the electronic device is an XR device, the rendering method includes: receiving data to be rendered obtained by the XR device; performing pre-rendering processing on the data to be rendered through the native monado runtime, and submitting the obtained rendering scene data to the streaming XR application; sending the rendering scene data to the computing device through the streaming XR application, causing the computing device to perform rendering processing, and sending the obtained rendering result to the XR device; receiving the rendering result sent by the computing device through the streaming XR application, and submitting the rendering result to the native monado runtime; and displaying the rendering result through the native monado runtime.

[0111] When the electronic device is a computing device, the rendering method includes: receiving rendering scene data sent by the XR device through the streaming monado runtime, and submitting the rendering scene data to the native XR application; rendering the rendering scene data through the native XR application to obtain a rendering result, and submitting the rendering result to the streaming monado runtime; sending the rendering result to the XR device through the streaming monado runtime, so that the XR device displays the rendering result.

[0112] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. This understanding states that the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0113] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the rendering methods provided by the above methods.

[0114] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is configured to execute the rendering methods provided by the above methods when the computer program is executed by a processor.

[0115] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0116] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A rendering method, characterized in that: Applied to XR devices, including: Receiving data to be rendered obtained by the XR device; Performing pre-rendering processing on the data to be rendered through the native monado runtime, and submitting the resulting rendered scene data to the streaming XR application; Sending the rendered scene data to a computing device through the streaming XR application, causing the computing device to perform rendering processing, and sending the obtained rendering result to the XR device; receiving, through the streaming XR application, a rendering result sent by the computing device, and submitting the rendering result to the native monado runtime; The rendered result is displayed by the native monado runtime.

2. The rendering method according to claim 1, wherein: The computing device includes a streaming monado runtime and a native XR application; wherein the streaming monado runtime is used to receive rendering scene data sent by the streaming XR application; The native XR application is used to render the rendering scene data to obtain a rendering result.

3. The rendering method according to claim 1, wherein: The sending the rendered scene data to a computing device through the streaming XR application includes: Through the streaming XR application, the rendered scene data is sent to the computing device via the web real-time communication WebRTC protocol.

4. The rendering method according to claim 1, wherein: Before performing pre-rendering processing on the data to be rendered by the native Monado runtime, the method further includes: Get the current application; The pre-rendering processing of the data to be rendered by the native monado runtime includes: Determining a target processing method based on the current application; When the native monado is running, the data to be rendered is pre-processed according to the target processing mode.

5. A rendering method, characterized in that: Applied to computing devices, including: Receive rendered scene data sent by the XR device through the streaming monado runtime, and submit the rendered scene data to the native XR application; Rendering the rendering scene data through the native XR application to obtain a rendering result, and submitting the rendering result to the streaming monado runtime; The rendering result is sent to the XR device through the streaming monado runtime, so that the XR device displays the rendering result.

6. The rendering method according to claim 5, characterized in that: The XR device includes a native monado runtime and a streaming XR application; wherein the native monado runtime is used to perform pre-rendering processing on the data to be rendered to obtain rendering scene data; The streaming XR application is used to send the rendered scene data to the computing device, and is also used to receive the rendering results sent by the computing device and submit them to the native monado runtime.

7. The rendering method according to claim 5, characterized in that: The step of sending the rendering result to the XR device through the streaming monado runtime includes: When the streaming monado is running, the rendering result is sent to the XR device through the WebRTC protocol.

8. The rendering method according to claim 5, characterized in that: The rendering process of the rendering scene data by the native XR application to obtain a rendering result includes: Obtaining a current application, and determining a data update scenario based on the current application; The native XR application is used to render the rendering scene data according to the data update scene to obtain a rendering result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the rendering method according to any one of claims 1 to 8 is implemented.

10. A rendering system, characterized in that: Includes XR devices and computing devices, including: The XR device is configured to receive the to-be-rendered data acquired by the XR device, perform pre-rendering processing on the to-be-rendered data through a native monado runtime to obtain rendering scene data, and send the rendering scene data to a computing device through a streaming XR application; The computing device is configured to receive the rendering scene data through a streaming monado runtime, render the rendering scene data through a native XR application to obtain a rendering result, and send the rendering result to the XR device through the streaming monado runtime; The XR device is configured to receive the rendering result through the streaming XR application and display the rendering result through the native monado runtime.

Images