Light ray tracing method and system

Abstract

A ray tracing method performed by an electronic device. The mesh addition module adds the game object to a ray tracing world class associated with the scene. The material adding module adds the material of the game object to the ray tracing world class. A light addition module adds a light configuration to a ray tracing world class. The rendering module renders a ray tracing effect of at least a portion of the game object in the scene based on the ray tracing world class and generates a stereoscopic view of the scene including the game object.

Description

[0001] Cross - reference to related applications

[0002] This application claims the priority benefit of U.S. Provisional Patent Application No. 63 / 419,422, filed on October 26, 2022, the entire content of which is incorporated herein by reference. Technical field

[0003] This disclosure relates to the field of artificial reality, and more particularly to ray - tracing methods and systems.

[0004] Description of related technologies

[0005] Technologies related to extended reality (XR), such as virtual reality (VR), augmented reality (AR), mixed reality (MR), etc., have achieved rapid development. A system implementing artificial reality technology may include a device that allows digital - generated virtual objects, such as 3D virtual objects, to be positioned in a 3D scene, or digital - generated virtual objects to be superimposed on an image of the real - world environment and co - exist with objects in the real - world environment.

[0006] Technical problems

[0007] Implementing ray - tracing in real - time applications, whether on a personal - computer - based VR device or an integrated VR device, generally requires powerful graphics processing unit (GPU) computing power, which makes it a very challenging task in most scenarios.

[0008] Integrating ray - tracing technology requires a large amount of coding work from developers or designers and often requires a specific environment to be successfully implemented and integrated. Unfortunately, there is currently no general user - friendly system or method.

[0009] Implementing real - time ray - tracing effects with limited computing resources requires optimization in stereoscopic vision, ray - tracing algorithms, and pipeline design. However, existing solutions often neglect the optimization aspects for real - time scenarios and integrated VR devices. Summary of the invention

[0010] The objective of this disclosure is to propose a ray - tracing method and system.

[0011] In a first aspect, an embodiment of the present invention provides a ray - tracing method executable in an electronic device, the method including:

[0012] Adding a game object to a ray - tracing world class associated with a scene through a grid addition module;

[0013] Add the material of the game object to the ray tracing world class through the material addition module;

[0014] Add the light configuration to the ray tracing world class through the light addition module;

[0015] Render a ray tracing effect for at least a part of the game objects in the scene based on the ray tracing world class through the rendering module; and

[0016] Generate a stereoscopic view of the scene including the game objects through the rendering module.

[0017] In a second aspect, an embodiment of the present invention provides an electronic device including a processor, which is configured to call and run a computer program stored in a memory, so that the device installed with the chip executes any combination of the disclosed method and the embodiments of the disclosed method.

[0018] In a third aspect, an embodiment of the present invention provides a ray tracing system, which includes:

[0019] A mesh addition module, configured to add a game object to a ray tracing world class associated with a scene;

[0020] A material addition module, configured to add the material of the game object to the ray tracing world class;

[0021] A light addition module, configured to add a light configuration to the ray tracing world class; and

[0022] A rendering module, configured to render a ray tracing effect for at least a part of the game objects in the scene based on the ray tracing world class, wherein the rendering module generates a stereoscopic view of the scene including the game objects.

[0023] The disclosed method can be programmed as computer-executable instructions stored in a non-transitory computer-readable medium. When the non-transitory computer-readable medium is loaded into a computer, it guides the processor of the computer to execute the disclosed method.

[0024] The non-transitory computer-readable medium may include at least one of a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory.

[0025] The disclosed method can also be programmed as a computer program product, which causes a computer to execute the disclosed method.

[0026] The disclosed method can also be programmed as a computer program that causes a computer to execute the disclosed method.

[0027] The disclosed system and method provide a real-time solution for rendering ray tracing effects for virtual reality, augmented reality, and mixed reality applications.

[0028] The disclosed system and method are applicable to integrated devices and AR / VR / MR devices based on PCs or smartphones.

[0029] The disclosed system is implemented as a lightweight plug-in that can be integrated into a game engine to provide ray tracing effects.

[0030] The optimization method can be enabled / disabled manually or automatically according to the scene complexity and computing resources. Brief Description of the Drawings

[0031] To more clearly illustrate the embodiments of the present disclosure or related technologies, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can also obtain other drawings based on these drawings without creative efforts.

[0032] Figure 1 A schematic diagram of an electronic device for extended reality (XR) is shown.

[0033] Figure 2 A schematic diagram of an example of a personal computer executing the disclosed system and method is shown.

[0034] Figure 3 A schematic diagram of an embodiment of the system of the present disclosure is shown.

[0035] Figure 4 A schematic diagram of an embodiment of the disclosed method is shown.

[0036] Figure 5 A schematic diagram of an example of a left view and a right view is shown.

[0037] Figure 6 A schematic diagram of an embodiment of the rendering plug-in of the present disclosure is shown.

[0038] Figure 7 A schematic diagram of another example of a left view and a right view in a multi-channel rendering scheme is shown.

[0039] Figure 8 A schematic diagram of another example of a left view and a right view in a multi-view rendering scheme is shown.

[0040] Figure 9 A schematic diagram of view synthesis using depth image-based rendering (DIBR) is shown.

[0041] Figure 10 A schematic diagram showing a first example of a scene processed by hybrid rendering that combines rasterization.

[0042] Figure 11 A schematic diagram showing a second example of reducing a reflection area based on a material.

[0043] Figure 12 A schematic diagram showing a chip that executes the disclosed method. Detailed implementation manners

[0044] The technical content, structural features, achieved objectives, and effects of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Specifically, the terms in the embodiments of the present disclosure are only used for the purpose of describing specific embodiments and do not limit the present disclosure.

[0045] Table 1

[0046] API Application Programming Interface AR Augmented Reality BVH Bounding Volume Hierarchy CPU Central Processing Unit DIBR Depth Image-Based Rendering FBO Frame Buffer Object GPU Graphics Processing Unit MR Mixed Reality ORM Object Relational Management PC Personal Computer SDK Software Development Kit VR Virtual Reality (VR) XR Extended Reality

[0047] The present disclosure provides a new framework that enables ray tracing to be implemented in XR. The system of the present disclosure includes a native ray tracing software development kit (SDK) designed for mobile devices or personal computers (PCs). For example, mobile devices may include smartphones, tablets, etc. Mobile devices may execute an embedded operating system, such as Android TM .

[0048] The system of the present disclosure may further include a plug-in that encapsulates native functions and exposes them to a game engine. The embodiments of the present disclosure provide methods for rendering stereoscopic vision in VR and optimizations required for processing complex scenes on VR devices.

[0049] The ray tracing SDK facilitates real-time ray tracing solutions for both desktop and mobile platforms. To enable VR to fully utilize the advantages of the ray tracing SDK, a rendering plug-in was developed to seamlessly integrate the static libraries of the SDK and expose script-accessible application programming interfaces (APIs) for invocation from within the game engine.

[0050] In addition, to meet the requirement that VR rendering needs to create and render dual views for both the left-eye camera and the right-eye camera, specific methods for generating stereoscopic vision were established. These methods focus on the creation of dual views to ensure an immersive VR experience.

[0051] To seamlessly integrate the system into real-time applications and ensure an excellent visual experience in the VR scene, several optimization techniques are adopted. These optimization techniques include hybrid rendering techniques combined with rasterization, reducing the size of shadow maps, restricting the reflection area based on physical properties and materials, and using mesh space rendering for static scenes, etc.

[0052] Refer to Figure 1 , the system includes an XR device 10a. The XR device 10a executes the method according to an embodiment of the present disclosure. The XR device 10a can be a mobile phone, a PC-based XR device, a stand-alone XR device, AR / VR glasses, or other XR processing devices. Figure 1 By way of example only and not limitation, the system can include more XR devices. Connections between devices and device components are represented by lines and arrows in the figure. The XR device 10a can include a processor 11a, a memory 12a, a transceiver 13a, a camera 14a, a depth camera 15a, and an inertial measurement unit (IMU) 16a. The camera 14a captures a scene and generates a color space image of the scene. The depth camera 15a captures a scene and generates a depth image of the scene. The IMU 16a measures and generates the external odometry of the device 10a. The odometry of the device is an estimate that uses data from motion sensors to estimate the change in the position of the device over time. The color space image camera (such as the camera 14a) is configured to capture a series of input frames, where each input frame includes a color space image. The depth camera (such as the depth camera 15a) is configured to capture a depth image associated with the color space image in each frame. The IMU (such as the IMU 16a) is configured to provide external odometry associated with the color space image in each frame. In various embodiments, the display 17a can include displays such as a liquid crystal display and a touch screen display. The display 17a can display a left view 141a and a right view to achieve a stereoscopic view for the user.

[0053] The processor 11a can include an application specific integrated circuit (ASIC), other chip sets, logic circuits, and / or data processing devices. The memory 12a can include a read only memory (ROM), a random access memory (RAM), a flash memory, a memory card, a storage medium, and / or other storage devices. The transceiver 13a can include a baseband circuit and a radio frequency (RF) circuit. When an embodiment is implemented in software, the techniques described herein can be implemented by modules, procedures, functions, entities, etc. that execute the functions described herein. These modules can be stored in the memory and executed by the processor. The memory can be implemented inside or outside the processor, and the memory can be communicatively coupled to the processor in various ways known in the art.

[0054] Refer to Figure 2, a personal computer (PC) 200 may include a processor 21a, a memory 22a, and a transceiver 23a. The processor 21a is used to call and run a computer program stored in the memory 22a, so that the PC 200 equipped with the processor 11 executes the methods, steps, and / or functions of one or more embodiments of the present disclosure. The transceiver 23a may include a baseband circuit and a radio frequency (RF) circuit.

[0055] The processor 21a may include an application specific integrated circuit (ASIC), other chip sets, logic circuits, and / or data processing devices. The memory 22a may include a read only memory (ROM), a random access memory (RAM), a flash memory, a memory card, a storage medium, and / or other storage devices. The transceiver 23a may include a network interface card (NIC) or a wireless communication unit, which may include a baseband circuit and a radio frequency (RF) circuit. When an embodiment is implemented in software, the technologies described herein may be implemented by modules, procedures, functions, entities, etc. that execute the functions described herein. These modules may be stored in the memory and executed by the processor. The memory may be implemented inside or outside the processor, and the memory may be communicatively coupled to the processor in various ways known in the art.

[0056] System architecture:

[0057] As Figure 3 shown, the VR ray tracing system 100 includes multiple modules, including a game engine 50, a rendering plug-in 51, an XRSDK 52, a scene 53, a custom shader, a native SDK 55, a static library 56, a shader 57, and a texture 58. The ray tracing system 100 may be installed on the XR device 10a or the PC 200 and executed by them.

[0058] The rendering plug-in 51 and the native SDK 55 may be seamlessly integrated into the game engine 50 or utilized by the game engine 50. The rendering plug-in 51 is configured to call native functions within the native SDK 55 for ray tracing. The native SDK 55 for ray tracing is configured to perform all necessary calculations required for effects such as shadows, reflections, and refractions.

[0059] The XR SDK 52 is operable to configure camera information and render scenes on the VR device, typically customized for a specific platform and equipped with hardware devices. Examples of the XR SDK 52 are, for example, Oculus TM Plugin and Pico TMXRSDK. Once all necessary dependencies for a game object are in place, the developer can design a scene (e.g., scene 53) in the game engine 50 by importing 3D assets or utilizing the built-in editing tools of the game engine 50, just as in the case of a non-ray tracing application or game. Ultimately, a custom shader 54 attached to the game object renders the shadow map onto the game object and applies the ray tracing effect to the original color of the game object.

[0060] See also Figure 4 , embodiments of the disclosed ray tracing method include:

[0061] Add the game object to the Ray Tracing World class associated with the scene via the Mesh Add module (B101);

[0062] Add the material of the game object to the ray tracing world class (B102) through the material adding module;

[0063] Added light configuration to the raytraced world class via the lightadd module (B103);

[0064] Rendering a ray tracing effect for at least a portion of the game object in the scene based on the ray tracing world class by a rendering module (B104); and

[0065] A stereoscopic view of the scene including the game objects is generated by a rendering module (B105).

[0066] like Figure 6 As shown, in some embodiments of the present disclosure, examples of a mesh adding module, a material adding module, a light adding module, and a rendering module include adding a mesh to a ray tracing world (addMeshToRTworld) 621, adding a material to a ray tracing world (addMaterialToRTworld) 622, adding a light to a ray tracing world (addLightToRTworld) 623, and rendering a shadow map (renderShadowMap) 624.

[0067] In some embodiments of the present disclosure, the mesh adding module, the material adding module, the light adding module and the rendering module are included in a software development kit (SDK). In some embodiments of the present disclosure, the SDK is included in a game engine.

[0068] In some embodiments of the present disclosure, the material of the game object includes albedo (reflection energy, white), normal, object relationship mapping (ORM), color, emission (emission, radiation), roughness and metallic. The light configuration includes a light source.

[0069] In some embodiments of the present disclosure, the stereoscopic view is generated in a multi-pass rendering mode, in which the game engine renders the scene twice for each game object using two draw calls.

[0070] In some embodiments of the present disclosure, the stereoscopic view is generated in a multi-view rendering mode, in which the game engine alternately renders the scene between the left view and the right view. The graphics processing unit (GPU) performs a single pass on all game objects in the scene for culling, and renders the game objects that successfully pass the culling process.

[0071] In some embodiments of the present disclosure, the stereoscopic view is generated in a depth image based rendering (DIBR) mode, in which the left view and the depth map are used as inputs, and the right view is generated through 3D warping and hole filling.

[0072] In some embodiments of the present disclosure, the optimizable rendering function can be enabled or disabled. In some embodiments of the present disclosure, the optimization function includes a hybrid method, in which a part of the shadow area in the scene is generated by rasterization, and another part of the shadow area in the scene is recalculated by a ray tracing method. In some embodiments of the present disclosure, the optimization function includes reflection area reduction, which includes:

[0073] determining whether the mesh of the game object is reflective; and

[0074] when the mesh of the game object is reflective, adding the mesh of the game object to the ray tracing world class, where a bounding volume hierarchy (BVH) is constructed for the mesh and the ray tracing effect is rendered for the mesh.

[0075] Product integration:

[0076] Refer to Figure 5 , which provides an example of the native SDK 55. The native plugin 620 is an example of the native SDK 55. This example can be integrating the VR ray tracing system with Unity TM and Oculus TMExamples of using together. In this scenario, the spheres 25 and the wine bottles 26 in the left view 24a and the right view 24b are set as reflective surfaces with different metallic luster and roughness settings. By parsing all this information of the spheres 25 and the wine bottles 26 into the native SDK 55, the reflections on the surfaces of the spheres 25 and the wine bottles 26 are calculated. For the shadows in this scenario, the system in the embodiments of the present disclosure also uses a hybrid method that combines ray-traced shadows and rasterized shadows to reduce complexity and computational load.

[0077] Rendering plug-in for game engines:

[0078] See Figure 6 , an embodiment of the rendering plug-in RenderingPlugin 51 is provided. RenderingPlugin 51 serves as an intermediate layer between the game engine 50 scene in the system 100 and the native ray-tracing function (e.g., the native SDK 55).

[0079] The model initializes the ray-tracing world (InitializeRTworld) 601 is a function for initializing the ray-tracing world in an XR application. The real-time world is a virtual environment or virtual scene that is rendered and updated according to the user's actions and inputs. This function takes some parameters that define the properties and settings of the real-time world, such as size, lighting, physical properties, and objects. This function also creates and returns a handle to the ray-tracing world, which can be used for subsequent access and modification.

[0080] The camera pre-render (onCamerapreRender) 602 is a function for executing some code before the camera represented by the camera object in an XR application starts to render. It is similar to the Camera.onPreRender event in Unity TM , which allows users to register callback functions that are called before any camera renders. The difference is that onCamerapreRender 602 is specific to each camera (e.g., camera 14a), while Camera.onPreRender is executed globally for all cameras.

[0081] onCamerapreRender 602 performs some operations that affect the appearance or behavior of the camera or the scene before the rendering process starts. For example, onCamerapreRender 602 can:

[0082] · Adjust camera parameters, such as the field of view, projection matrix, or clipping plane.

[0083] · Modify scene objects, such as changing their position, rotation, scale, or material.

[0084] · Apply some effects, such as lens distortion, chromatic aberration, or vignetting.

[0085] UpdateGameObject 610 is a function for updating the properties and behaviors of game objects (GameObjects) in an XR application. A GameObject is the basic unit of a scene, which can represent characters, props, scenery, cameras, etc. The functions of a GameObject are defined by components attached to it, such components being, for example, scripts, renderers, colliders, etc.

[0086] The UpdateGameObject 610 function takes a GameObject as an argument and performs some operations on it, such as changing its position, rotation, scale, material, or animation. The UpdateGameObject 610 function can be called by onCameraPreRender 602. Alternatively, in a script attached to the GameObject, the Update method can call the UpdateGameObject 610 function every frame to make the GameObject move, rotate, or animate according to some logic or input.

[0087] UpdateCamera 611 is a function for updating the properties and behaviors of cameras in an XR application. A camera is a component that captures and displays a scene from a certain viewpoint. The functions of a camera are defined by parameters attached to it, such parameters being, for example, field of view, projection mode, clipping planes, and target texture.

[0088] The UpdateCamera 611 function takes a camera as an argument and performs some operations on it, such as changing its position, rotation, scale, or focus. The UpdateCamera 611 function can be called by UpdateGameObject 610. Alternatively, the UpdateCamera 611 function can be called every frame through the Update method in a script attached to the camera to make the camera follow, view, or rotate around a target object according to some logic or input.

[0089] m_ShadowMap.Render 612 is a function for rendering a shadow map in a game engine. A shadow map is a texture that stores the depth values of a scene from the viewpoint of a light. By comparing the depth values of the scene from the camera's viewpoint with the depth values of the shadow map, the shadow map can be used to create realistic shadows.

[0090] The m_ShadowMap.Render 612 function takes a light source and a scene as parameters and performs the following steps:

[0091] · Create a frame buffer object (FBO) and attach a depth texture to it. The FBO is used to render an off-screen scene and store depth values in the texture.

[0092] · Set the viewport size and projection matrix according to the parameters of the light source, such as position, direction, and angle. The projection matrix defines how the scene is projected onto the texture.

[0093] · Bind the FBO and clear the depth buffer. Also enable depth testing and front-face culling to avoid self-shadowing artifacts.

[0094] · Render the scene using a shader that outputs only the depth value of each fragment. The shader can also apply some biases or offsets to avoid shadow acne or light leaks.

[0095] · Unbind the FBO and restore the original viewport size and projection matrix. Thus, the depth texture is ready for shadow mapping.

[0096] The m_ShadowMap.Render 612 function implements shadow mapping. Different game engines may have different ways of rendering shadow maps, but the basic principle is similar.

[0097] renderShadowMap 624 can be used to render shadows. This rendering involves creating a texture (called a shadow map) that stores the depth values of the scene from the perspective of the light source. Then, in the final rendering pass, the shadow map is used to determine whether a pixel is in shadow by comparing the depth value of the pixel with the depth value stored in the shadow map. renderShadowMap 624 can create realistic dynamic shadows for various types of scenes and objects (such as trees, buildings, people, etc.).

[0098] renderShadowMap 624 uses a stereoscopic rendering mode that creates two images, one for each eye, with a slight horizontal offset between the two images to simulate the distance between the eyes. There are mainly three modes of stereoscopic rendering:

[0099] (1) Multi-pass rendering;

[0100] (2) Multi-view rendering; and

[0101] (3) DIBR-based view synthesis.

[0102] Standard ORM shaders are commonly used in modern game engines and 3D modeling tools that support physically based rendering (PBR) materials. PBR materials are materials that simulate how light interacts with real-world materials in a realistic way.

[0103] The standard ORM shader 641 encodes the values of occlusion, roughness, and metallicity by using different color channels of a texture. The red channel stores occlusion, which is the amount of ambient light that reaches the surface. The green channel stores roughness, which is the smoothness or roughness of the surface. The blue channel stores metallicity, which is the metallic or non-metallic nature of the surface.

[0104] The advantage of using the standard ORM shader 641 is that it reduces the number of textures required for a material, which can improve the performance and memory usage of an application. It also makes file management easier because there is only one texture file for each material.

[0105] The camera rendering module 642 uses the output of the standard ORM shader 641 to create realistic and immersive images for XR applications.

[0106] The ray tracing world initialization module InitializeRTWorld 601 can initialize game objects associated with the ray tracing world class RTWorld. Each game object is represented by the object GameObject. Scripts attached to the game object (such as onCameraPreRender 602) use modules in the native plugin 620 to iteratively add information about the object, camera, material, and light to the RTWorld class. The native plugin 620 is included in the native SDK 55. This information includes the position matrix, the properties of the object, and the information required for ray tracing calculations. For example, the camera pre-render module onCameraPreRender 602 calls the module addMeshToRTworld 621 to add an object to the RTWorld class, calls the module addMaterialToRTworld 622 to add a material to the RTWorld class, and calls the module addLightToRTworld623 to add light to the RTWorld class. The module addMaterialToRTworld 622 can use material properties such as albedo, normal, ORM, color, emission, roughness, and metallicity from the material library 650.

[0107] The game object update module UpdateGameObject 610 updates game objects. The camera update module UpdateCamera611 uses the game objects to update the camera 14a.

[0108] Two custom shadow maps will be created for the left view and the right view respectively as the rendering targets for ray tracing effects (such as shadows, reflections, refractions, etc.). These two custom shadow maps include the RayTracedShadowMap 631 for the left view 141a and the RayTracedShadowMap 632 for the right view 142a. The shadow map rendering module m_ShadowMap.Render 612 calls the module renderShadowMap 624 to generate the RayTracedShadowMap 631 for the left view and the RayTracedShadowMap 632 for the right view by calling the native functions in the ray tracing SDK (i.e., the native SDK 55) that perform all the calculations for ray tracing effects (such as shadows, reflections, refractions, etc.). The standard object-relational mapping (ORM) shader 641 adds the shadow maps to the original scene (such as RTWorld) to add all the ray tracing effects.

[0109] Stereo vision:

[0110] The native plugin 620 performs rendering to generate the left view and the right view. The native plugin 620 can drive the GPU for rendering.

[0111] (1) Multi-channel rendering:

[0112] Refer to Figure 7 , and the embodiments of the present disclosure using multi-channel rendering are described in detail below. The game engine 50 renders the scene (such as scene 53) twice using 2 draw calls for each game object (i.e., GameObject) with a Renderer component. When rendering both the left view and the right view (such as the left view 241a and the right view 241b), the Renderer component only traverses the scene Figure 1 once. Ray tracing effects are rendered on two shadow maps for each eye (such as the RayTracedShadowMap 631 for the left view and the RayTracedShadowMap 632 for the right view). In the multi-channel rendering scheme, the two shadow maps will be rendered to the left eye or the right eye in sequence.

[0113] (2) Multi-view rendering:

[0114] Refer to Figure 8, embodiments of the present disclosure using multi - view rendering are described in detail below. During multi - view rendering, the left view and the right view (e.g., left view 242a and right view 242b) share the work required for culling and shadow calculations. Culling (e.g., frustum culling, occlusion culling, and level of detail (LOD) culling) reduces the number of objects to be rendered in the scene by discarding objects that are not visible to the camera. In a multi - view rendering scheme, the graphics processing unit (GPU) renders each game object (i.e., GameObject) in a ping - pong manner, in which the game objects are alternately rendered between the two eyes. Thus, there are fewer graphics commands to change or switch states. The GPU makes a single pass through all the game objects GameObject in the scene for culling processing and renders the GameObjects that successfully pass the culling process. For the ray - traced shadow map of the scene, the left - eye view and the right - eye view (e.g., left view 242a and right view 242b) are clipped to a render texture, and the left and right halves of the render texture are used for the views of each eye respectively. Then, the camera rendering process (i.e., camera renderer 642) will only acquire the combined texture once and render it onto the game object through a single draw call.

[0115] (3) DIBR - based view synthesis:

[0116] Refer to Figure 9 , embodiments of the present disclosure using a DIBR - based view synthesis scheme are described in detail below. To further reduce the computational load of generating two shadow maps for the left - eye view and the right - eye view, a DIBR - based view synthesis scheme can be adopted. DIBR stands for depth - image - based rendering, which uses the left view (e.g., left view 243a) and the depth map (e.g., depth map 243) as inputs and generates the right view (e.g., right view 243b) through 3D warping and hole filling (e.g., 3D warping 661 and hole filling 662). This reduces the ray - tracing calculation by half and can handle objects that are not close to the viewpoint well.

[0117] Rendering optimization:

[0118] (1) Hybrid rendering combined with rasterization:

[0119] Refer to Figure 10, Embodiments of the present disclosure using a DIBR-based view synthesis scheme are described in detail below. The CPUs and GPUs on current VR devices are not as powerful as those on smartphones, let alone the latest GPUs with dedicated hardware for ray tracing calculations. Therefore, the system must be simplified and adjusted to achieve the goal of real-time ray tracing for virtual reality scenarios. For example, some ray tracing effects may be partially or fully disabled to improve efficiency. In this system, embodiments of the present disclosure provide a hybrid method for rendering shadows. In this hybrid method, most of the shadow areas in the scene are generated by rasterization. The shadow areas generated by rasterization are only hard shadows, such as Figure 10 the black part in scene 244 of

[0120] (2) Reducing the reflection area based on the material:

[0121] Refer to Figure 11 , Embodiments of the present disclosure for reducing the reflection area based on the material are described in detail below. These operations can be performed by the native plugin 620. To enhance the ray tracing performance on VR devices, non-reflective game objects will not be added to the RT world, thus omitting non-reflective objects from ray tracing calculations. Therefore, most of the background scenes or objects remain unchanged in their original settings. The reflection situation of each game object can be determined by the material of the object or further calculated based on the physical characteristics of the object. For example, only the most metallic and smoothest areas on a bottle will be considered reflective.

[0122] As Figure 11 shown, steps 711 and 712 can be applied to each game object. As the current game object, the input mesh 710 is input into the system 100. The system 100 determines whether the mesh 710 is reflective (711). If the mesh 710 is not reflective, the system 100 inputs the mesh 710 for camera rendering (715). If the mesh 710 is reflective, the system 100 adds the mesh 710 to the class RTWorld (712). The system 100 constructs a BVH for the mesh 710 (713) and performs ray tracing rendering on the mesh 710 (714). The system 100 performs camera rendering on the game object (715).

[0123] Figure 11 The operations in Figure 6 can be performed in the GPU or can be performed in the

[0124] In some embodiments of the present disclosure, ray tracing can also be completed entirely within a game engine (such as game engine 50) without using a native plugin (such as RenderingPlugin 51) and invoking native functions (such as native SDK 55).

[0125] In some embodiments of the present disclosure, a native plugin (such as RenderingPlugin 51) and native functions (such as native SDK 55) can be integrated into a game engine (such as game engine 50), or can be used as a third-party library in an application to run ray tracing effects.

[0126] In some embodiments of the present disclosure, stereoscopic vision can be configured to be left-eye dominant, right-eye dominant, or centered-view dominant to achieve real-time ray tracing effects on AR / VR / MR devices.

[0127] In DIBR-based view synthesis, the left view can be used to generate the right view. Alternatively, the right view can also be used to generate the left view, or the centered view can be used to generate the left view or the right view.

[0128] Referring to Figure 12 , embodiments of the present disclosure also provide a chip 700, which can correspond to the XR device 10a in the embodiments of the present disclosure. The chip 700 can implement the corresponding methods implemented by the XR device 10a in the method embodiments of the present disclosure. The chip 700 includes a processor 701, and the processor 701 can call and run a computer program in a memory to implement the methods of the embodiments of the present application.

[0129] Optionally, the chip 700 may further include a memory 702. In particular, the processor 701 can call and run a computer program in the memory 702 to implement the methods of the embodiments of the present application.

[0130] In addition, the memory 702 can be a device separate from the processor 701, or can be integrated into the processor 701.

[0131] Optionally, the chip 700 may further include an input interface 703. It should be noted that the processor 701 can control the input interface 703 to communicate with other devices or chips, specifically, to obtain messages or data sent by other devices or chips.

[0132] Optionally, the chip 700 may further include an output interface 704. It should be noted that the processor 701 can control the output interface 704 to communicate with other devices or chips, specifically, to output messages or data to other devices or chips.

[0133] The described system and method provide a real-time solution for rendering ray tracing effects for virtual reality, augmented reality, and mixed reality applications.

[0134] The system and method are applicable to integrated devices as well as PC- or smartphone-based AR / VR / MR devices.

[0135] The system is implemented as a lightweight plug-in that can be integrated into a game engine (such as game engine 50) to provide ray tracing effects.

[0136] Depending on the scene complexity and computing resources, the optimization method can be enabled or disabled manually or automatically.

[0137] Although the present disclosure has been described in connection with what are considered to be the most practical and preferred embodiments, it is to be understood that the present disclosure is not limited to the disclosed embodiments, but is intended to cover various arrangements made without departing from the scope of the broadest interpretation of the appended claims.

Claims

1. A ray tracing method executed by an electronic device, comprising: adding a game object to a ray tracing world class associated with a scene through a mesh addition module; adding the material of the game object to the ray tracing world class through a material addition module; adding a light configuration to the ray tracing world class through a light addition module; rendering a ray tracing effect for at least a part of the game object in the scene based on the ray tracing world class through a rendering module; and generating a stereoscopic view of a scene including the game object through the rendering module.

2. The ray tracing method according to claim 1, wherein, the mesh addition module, the material addition module, the light addition module, and the rendering module are included in a software development kit (SDK).

3. The ray tracing method according to claim 2, wherein, the SDK is included in a game engine.

4. The ray tracing method according to any one of claims 1 to 3, wherein, the material of the game object includes albedo, normal, object-relational mapping (ORM), color, emission, roughness, and metallicity.

5. The ray tracing method according to any one of claims 1 to 4, wherein, the light configuration includes a light source.

6. The ray tracing method according to any one of claims 1 to 5, wherein, the stereoscopic view is generated in a multi-channel rendering mode, in which the game engine renders the scene twice for each game object using two draw calls.

7. The ray tracing method according to any one of claims 1 to 6, wherein, the stereoscopic view is generated in a multi-view rendering mode, in which the game engine alternately renders the scene between a left view and a right view.

8. The ray tracing method according to any one of claims 1 to 7, wherein, a graphics processing unit (GPU) performs a single pass on all game objects in the scene for culling processing and renders the game objects that successfully pass the culling processing.

9. The ray tracing method according to any one of claims 1 to 8, wherein, the stereoscopic view is generated in a depth image-based rendering (DIBR) mode, in which a left view and a depth map are used as inputs, and a right view is generated through 3D warping and hole filling.

10. The ray tracing method according to any one of claims 1 to 9, wherein, the optimization function of the rendering can be enabled or disabled.

11. The ray tracing method according to any one of claims 1 to 10, wherein, the optimization function includes a hybrid method, in which part of the shadow area in the scene is generated through rasterization, and another part of the shadow area in the scene is recalculated through the ray tracing method.

12. The ray tracing method according to any one of claims 1 to 10, wherein, the optimization function includes reflection area reduction, and the reflection area reduction includes: determining whether the mesh of the game object is reflective; and When the mesh of the game object is reflective, add the mesh of the game object to the ray tracing world class, where a bounding volume hierarchy (BVH) is constructed for the mesh and ray tracing effects are rendered for the mesh.

13. An electronic device, comprising: a processor configured to call and run a computer program stored in a memory, so that the device installed with the processor executes the method according to any one of claims 1 to 12.

14. A chip, comprising: a processor configured to call and run a computer program stored in a memory, so that the device installed with the chip executes the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored therein, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 12.

17. A computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 12.

18. A ray tracing system, comprising: a mesh addition module configured to add a game object to a ray tracing world class associated with a scene; a material addition module configured to add the material of the game object to the ray tracing world class; a light addition module configured to add a light configuration to the ray tracing world class; and a rendering module configured to render ray tracing effects for at least a part of the game object in the scene based on the ray tracing world class, wherein the rendering module generates a stereoscopic view of the scene including the game object.

19. The ray tracing system according to claim 18, wherein, the mesh addition module, the material addition module, the light addition module, and the rendering module are included in a software development kit (SDK).

20. The ray tracing system according to claim 19, wherein, the SDK is included in a game engine.

21. The ray tracing system according to any one of claims 18 to 20, wherein, the material of the game object includes albedo, normal, object-relational mapping (ORM), color, emission, roughness, and metallicity.

22. The ray tracing system according to any one of claims 18 to 21, wherein, the light configuration includes a light source.

23. The ray tracing system according to any one of claims 18 to 22, wherein, the stereoscopic view is generated in a multi-channel rendering mode, in which the game engine renders the scene twice for each game object using two draw calls.

24. The ray tracing system according to any one of claims 18 to 23, wherein, the stereoscopic view is generated in a multi-view rendering mode, in which the game engine alternately renders the scene between a left view and a right view.

25. The ray tracing system according to any one of claims 18 to 24, wherein, the graphics processing unit (GPU) performs a single pass on all game objects in the scene for culling processing and renders the game objects that successfully pass the culling processing.

26. The ray tracing system according to any one of claims 18 to 25, wherein, the stereoscopic view is generated in a depth image-based rendering (DIBR) mode, and in the depth image-based rendering mode, the left view and the depth map are used as inputs, and the right view is generated through 3D wrapping and hole filling.

27. The ray tracing system according to any one of claims 18 to 26, wherein, the optimization function of the rendering can be enabled or disabled.

28. The ray tracing system according to any one of claims 18 to 27, wherein, the optimization function includes a hybrid method, in which part of the shadow area in the scene is generated by rasterization, and another part of the shadow area in the scene is recalculated by a ray tracing method.

29. The ray tracing system according to any one of claims 18 to 27, wherein, the optimization function includes reflection area reduction, and the reflection area reduction includes: determining whether the mesh of the game object is reflective; and when the mesh of the game object is reflective, adding the mesh of the game object to the ray tracing world class, wherein a bounding volume hierarchy (BVH) is constructed for the mesh and the rendering of the ray tracing effect is performed for the mesh.

Images