Method, device, equipment and product for drawing object

Abstract

The invention relates to a method, a device, equipment and a product for drawing an object. The method includes drawing a first object in a drawing space of an augmented reality environment, at least a portion of the first object being opaque. The method further includes determining a first portion of the second object occluded by the first object based on hierarchical information of the first object. In addition, the method further comprises the step of drawing a second part, not shielded by the first object, in the second object under the condition that the first part of the second object does not participate in drawing.

Description

Technical Field

[0001] This disclosure relates to the field of extended reality, and more specifically to methods, apparatus, devices, and products for drawing objects. Background Technology

[0002] Extended reality (XR) refers to an interactive environment that combines real and virtual elements, created through computer technology and wearable devices. Extended reality is a collective term for various technologies, including augmented reality (AR), virtual reality (VR), and mixed reality (MR). In extended reality devices, the rendering system blends virtual objects with the real environment to display the merged scene on the device's screen.

[0003] To present high-resolution, low-latency visuals in real time on head-mounted display devices (such as MR headsets or AR glasses), a spatial operating system can be used to manage and schedule the position, orientation, and drawing order of multiple virtual objects in three-dimensional space. Summary of the Invention

[0004] In a first aspect of the embodiments of this disclosure, a method for drawing an object is provided. The method includes drawing a first object in a drawing space of an extended reality environment, at least a portion of the first object being opaque. The method further includes determining a first portion of a second object that is occluded by the first object based on hierarchical information of the first object. Furthermore, the method includes drawing a second portion of the second object that is not occluded by the first object, provided that the first portion of the second object is not involved in the drawing.

[0005] In a second aspect of the embodiments of this disclosure, an apparatus for drawing an object is provided. The apparatus includes a first object drawing module configured to draw a first object in a drawing space of an extended reality environment, at least a portion of the first object being opaque. The apparatus also includes an occlusion portion determination module configured to determine a first portion of a second object occluded by the first object based on hierarchical information of the first object. Furthermore, the apparatus includes a second object drawing module configured to draw a second portion of the second object that is not occluded by the first object, provided that the first portion of the second object is not involved in the drawing.

[0006] In a third aspect of embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method for drawing an object. The method includes drawing a first object in a drawing space of an extended reality environment, at least a portion of the first object being opaque. The method further includes determining a first portion of a second object occluded by the first object based on hierarchical information of the first object. Furthermore, the method includes drawing a second portion of the second object that is not occluded by the first object, provided that the first portion of the second object is not involved in the drawing.

[0007] In a fourth aspect of embodiments of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to implement a method for drawing an object. The method includes drawing a first object in a drawing space of an extended reality environment, at least a portion of the first object being opaque. The method also includes determining a first portion of a second object occluded by the first object based on hierarchical information of the first object. Furthermore, the method includes drawing a second portion of the second object that is not occluded by the first object, provided that the first portion of the second object is not involved in the drawing.

[0008] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;

[0011] Figure 2 A flowchart of a method for drawing an object according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrating an example of drawing an object using a template buffer according to some embodiments of the present disclosure is shown;

[0013] Figure 4A schematic diagram illustrating an example of adjusting the drawing order of multiple objects in multiple drawing groups according to some embodiments of the present disclosure;

[0014] Figure 5 A schematic diagram illustrating an example of drawing multiple objects in multiple drawing groups according to some embodiments of the present disclosure;

[0015] Figure 6 A schematic diagram illustrating an example of drawing a feathered object according to some embodiments of the present disclosure is shown;

[0016] Figure 7 A block diagram of an apparatus for drawing objects according to some embodiments of the present disclosure is shown;

[0017] Figure 8 A block diagram of a device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0018] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0021] In traditional rendering, the rendering engine first determines the current visible area based on the user's perspective and sensor data, and then projects multiple virtual objects in the scene onto the eye buffer for pixel-by-pixel rendering. Extended reality scenes typically contain a large number of virtual elements superimposed with real elements, such as virtual panels in the foreground, semi-transparent objects, and 3D models in the background. Therefore, the system needs to complete the shading calculations for a large number of pixels in a very short time.

[0022] In some related technologies, mechanisms such as depth testing or low-resolution Z-squared (LRZ) testing can be used to cull occluded pixels by comparing the depth values ​​of multiple pixels, thereby ensuring visual accuracy. However, in these technologies, even if a pixel is occluded by an object in front, the system still calculates rendering information such as color, lighting, and texture sampling for that pixel. Only after the rendering information calculation is complete will the GPU compare the pixel's depth value with the current value in the depth buffer. If the pixel's depth value is large, meaning it is visually far from the user, the pixel is culled, resulting in a waste of time and resources used to calculate the rendering information for that pixel.

[0023] Furthermore, in MR / XR / AR device rendering, a Spatial OS is typically used to render complex scenes across the entire visual space into an eye buffer. A key characteristic of Spatial OS rendering is that the depth relationships between all rendered objects in the scene are dynamic, to meet the needs of dynamically changing Spatial OS rendering priorities. In this scenario, the global depth buffer needs to be frequently and dynamically modified, rendering traditional depth-based occlusion culling methods ineffective. For example, depth testing or LRZ testing will fail due to the frequent modification of pixel depth values, leading to decreased rendering efficiency and increased device power consumption.

[0024] Therefore, embodiments of this disclosure provide a scheme for drawing objects. In this scheme, a processing device can draw a first object in the drawing space of an extended reality environment, wherein at least a portion of the first object is opaque. Then, the processing device can determine a first portion of a second object that is occluded by the first object based on the hierarchy information of the first object rather than depth information. Then, the processing device can draw a second portion of the second object that is not occluded by the first object without the first portion of the second object participating in the drawing.

[0025] By using hierarchical information instead of depth values ​​to determine occlusion relationships between objects, pixels occluded by the first object can be removed when drawing a second object. This reduces the amount of computation required for drawing invisible pixels, decreases the amount of computation on invalid drawing information, improves the efficiency of the drawing process, and reduces waste of processing resources. Furthermore, using hierarchical information to determine occlusion relationships allows for accurate identification of occlusion areas even when the depth of objects dynamically changes, in scenarios such as multi-window overlays or dynamic user interaction panels, without recalculating pixel depth values, thus further reducing waste of processing resources.

[0026] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of this disclosure may be implemented is shown. For example... Figure 1 As shown, environment 100 includes processing device 102. Processing device 102 can be any device with processing or computing capabilities. For example, processing device 102 can be a mixed reality (MR) device, virtual reality (VR) device, augmented reality (AR) device, smartphone, desktop computer, laptop computer, tablet computer, personal assistant, or smart wearable device, etc.

[0027] In environment 100, processing device 102 can draw object 106 in the rendering space 104 of the extended reality environment, with at least a portion of object 106 being opaque. For example, the MR device can blend real-world scenes with virtual objects in the rendering space 104 and present the blended image to the user. Object 106 can be, for example, an application window, a system pop-up, an interactive component, etc. For example, object 106 can be a browser window, an instant messaging application window, or a content publishing platform window, etc. For example, when a user interacts with an application, the application's window can be placed on top of the rendering space 104, obscuring the windows of other applications.

[0028] In environment 100, processing device 102 can determine the portion 112 of object 108 that is occluded by object 106 based on the hierarchy information 110 of object 106. For example, object 108 may be a window of another application, and the display area of ​​object 108 overlaps with the display area of ​​object 106, such that object 106 occludes at least a portion of object 108 when object 106 is at a higher hierarchy than object 108. In some embodiments, hierarchy information 110 may be a value indicating the hierarchy of object 106 (e.g., 125, 127, etc.). In some embodiments, hierarchy information 106 may be a marker indicating whether object 106 will occlude other objects (e.g., "yes" or 1, etc.). In some embodiments, processing device 102 can assign hierarchy information 110 to multiple objects based on their visual proximity in drawing space 104. In some embodiments, when drawing object 108, processing device 102 can determine that the hierarchy information 110 of object 106 indicates that object 106 will occlude other objects. Then, the processing device 102 can determine the multiple pixels in object 108 that overlap with object 106 as the occluded portion 112. In some embodiments, when drawing object 108, the processing device 102 can obtain the layer information of object 108 and compare the layer information with the layer information 110 of object 106 to determine that the layer of object 108 is lower than (or farther than) the layer of object 106. Then, the processing device 102 can determine the multiple pixels in object 108 that overlap with object 106 as the occluded portion 112.

[0029] In environment 100, the processing device can draw the portion 114 of object 108 that is not occluded by object 106 without the portion 112 of object 108 participating in the drawing. For example, since the processing device 102 has determined that the portion 112 of object 108 is occluded by object 106, the processing device 102 can skip the calculation process associated with drawing the portion 112. For example, the processing device 102 can skip the fragment shading process for the portion 112, thereby no longer performing operations such as color calculation, lighting calculation, texture sampling, reflection or refraction calculation for the pixels in the portion 112. In addition, the processing device 102 can also skip the depth testing process for the portion 112, thereby no longer recalculating the depth of the pixels in the portion 112 and comparing it with the corresponding value in the depth buffer. Furthermore, the processing device 102 can also skip the color blending process for the portion 112, thereby no longer performing operations such as color blending or transparent overlay with background pixels.

[0030] In this way, by using the hierarchical information 110 of object 106 instead of the pixel depth values ​​of objects 106 and 108 to determine the occlusion relationship between objects, pixels in the portion 112 occluded by object 106 can be removed when drawing object 108. This reduces the amount of calculation of drawing information for invisible pixels by the processor 102, reduces the amount of calculation of invalid drawing information, improves the efficiency of the drawing process, and reduces the waste of processing resources. In addition, using the hierarchical information 110 to determine the occlusion relationship between objects can correctly determine the occlusion area even when the depth of objects 106 and 108 changes dynamically in scenarios such as multi-window overlay or dynamic user interaction panels, without having to recalculate the pixel depth values, thereby reducing the waste of processing resources.

[0031] Figure 2 A flowchart of a method 200 for drawing an object according to some embodiments of the present disclosure is shown. Method 200 can be executed by a processing device. For example, method 200 can be performed by… Figure 1 The processing device 102 in the middle performs the operation. For example... Figure 2 As shown in box 202, the processing device can draw a first object in the drawing space of the extended reality environment, at least a portion of the first object being opaque. For example, in... Figure 1 In the illustrated environment 100, the processing device 102 can draw objects 106 in the rendering space 104 of the extended reality environment, with at least a portion of the objects 106 being opaque. For example, the MR device can blend real-world scenes with virtual objects in the rendering space 104 and present the blended image to the user. Object 106 can be, for example, an application window, a system pop-up, an interactive component, etc. For example, object 106 can be a browser window, an instant messaging application window, or a content publishing platform window, etc. For example, when a user interacts with an application, the application's window can be placed on top of the rendering space 104, obscuring the windows of other applications.

[0032] In box 204, the processing device can determine the first portion of the second object that is occluded by the first object based on the hierarchy information of the first object. For example, in... Figure 1In the illustrated environment 100, processing device 102 can determine the portion 112 of object 108 that is occluded by object 106 based on the hierarchy information 110 of object 106. In some embodiments, when drawing object 108, processing device 102 can determine that the hierarchy information 110 of object 106 indicates that object 106 will occlude other objects. Then, processing device 102 can determine multiple pixels in object 108 that overlap with object 106 as the occluded portion 112. In some embodiments, when drawing object 108, processing device 102 can acquire the hierarchy information of object 108 and compare it with the hierarchy information 110 of object 106 to determine that the hierarchy of object 108 is lower than (or farther than) the hierarchy of object 106. Then, processing device 102 can determine multiple pixels in object 108 that overlap with object 106 as the occluded portion 112.

[0033] In box 206, the processing device can draw the second part of the second object that is not obscured by the first object, without the first part of the second object participating in the drawing. For example, in... Figure 1 In the environment 100 shown, the processing device can draw the portion 114 of object 108 that is not occluded by object 106 without the portion 112 of object 108 participating in the drawing. For example, since the processing device 102 has determined that the portion 112 of object 108 is occluded by object 106, the processing device 102 can skip the calculation process associated with drawing the portion 112.

[0034] By using hierarchical information instead of depth values ​​to determine occlusion relationships between objects, pixels occluded by the first object can be removed when drawing a second object. This reduces the amount of computation required for drawing invisible pixels, decreases the amount of computation on invalid drawing information, improves the efficiency of the drawing process, and reduces waste of processing resources. Furthermore, using hierarchical information to determine occlusion relationships allows for accurate identification of occlusion areas even when the depth of objects dynamically changes, in scenarios such as multi-window overlays or dynamic user interaction panels, without recalculating pixel depth values, thus further reducing waste of processing resources.

[0035] In some embodiments, when determining the first portion of a second object occluded by the first object based on the layer information of the first object, the processing device can update the stencil buffer corresponding to the drawing space based on the layer information of the first object. The stencil buffer contains layer information of at least a portion of the pixels in the drawing space. Then, the processing device can determine the first portion of the second object occluded by the first object based on the updated stencil buffer. By using the stencil buffer to record the layer information of each pixel in the drawing space, the occlusion determination process can be completed quickly at the hardware level. Furthermore, using the stencil buffer reduces the recalculation of the global depth buffer, saving computational resources while maintaining the accuracy of occlusion determination.

[0036] Figure 3 A schematic diagram of example 300, illustrating the use of a template buffer to draw objects according to some embodiments of the present disclosure, is shown. Figure 3 As shown, Example 300 includes an eye buffer 302, which includes world space and an opaque object 304. During the drawing process, the processing device can draw a standard view of each eye into the eye buffer 302, and then provide the eye buffer 302 as a drawing texture to the asynchronous time-warped (ATW) thread for distortion and sampling.

[0037] In Example 300, at step 1, the processing device can identify one or more opaque objects among a plurality of objects to be drawn. For each identified opaque object, the processing device can write to the stencil buffer 306 while writing to the depth buffer. The stencil buffer 306 allocates integer values ​​in pixels and shares a video memory area with the depth buffer. For example, each pixel can occupy one byte of storage space, allowing each pixel to be assigned a stencil value ranging from 0 to 255. For example, the processing device can assign a stencil value of "127" to opaque object 304. When writing to the stencil buffer 306, the processing device can write the stencil value 308 of opaque object 304 to the area in the stencil buffer 306 corresponding to opaque object 304; this stencil value can represent the hierarchical information of opaque object 304.

[0038] In step 2, for objects occluded by opaque objects, the processing device can discard pixels within that object that are obscured by the opaque object 304. For example, when the processing device is preparing to draw other objects obscured by the opaque object 304, it can initiate a stencil test. The processing device can determine whether each pixel belongs to the area obscured by the opaque object 304 based on the stencil value in the stencil buffer. If the stencil value corresponding to a pixel indicates that the pixel is obscured, the processing device can discard the pixel so that it is not involved in the drawing calculation process. If the stencil value corresponding to a pixel indicates that the pixel is not obscured, the processing device can continue to perform the drawing calculation process for that pixel. In this way, since the rendering occurs before the pixel shading stage, the processing device will no longer perform the drawing information calculation process for invisible pixels, thereby improving the drawing efficiency.

[0039] In step 3, after culling the occluded pixels based on the stencil buffer 306, the processing device can draw the opaque object 304' into the occlusion region 310 of the eye buffer 302'. At this time, the opaque object 304' can be correctly displayed in front of all the objects it occludes, thereby ensuring the consistency of the drawing order with the visual occlusion relationship.

[0040] In some embodiments, after discarding occluded pixels, the processing device can also perform an Early-Z depth test on the remaining pixels for secondary optimization. The Early-Z depth test can discard occluded pixels as early as possible before rasterization based on the pixel's depth value, thereby reducing the number of times the pixel shader is executed. By combining stencil testing and early-Z depth testing, rendering efficiency can be improved and processing resource consumption reduced in dynamic rendering scenarios such as Spatial OS.

[0041] In some embodiments, the hierarchy information is template values. The processing device can assign multiple template values ​​to multiple drawing groups based on their visual hierarchy, where a first drawing group includes a first object, and a second drawing group includes a second object. The processing device can draw opaque objects in the multiple drawing groups based on the multiple template values. The processing device can draw transparent or semi-transparent objects in the multiple drawing groups after all opaque objects in the multiple drawing groups have been drawn. In this way, the processing device can assign template values ​​according to the visual hierarchy of multiple drawing groups, thereby enabling unified drawing based on template values ​​across multiple logical layers (e.g., multiple application windows, system pop-ups, interactive components, etc.). Furthermore, through the collaborative drawing of opaque and transparent objects, the occlusion conflict problem when multiple windows are overlaid in dynamic drawing scenarios such as Spatial OS can be resolved, ensuring that the drawing results conform to the visual hierarchy logic.

[0042] In some embodiments, multiple drawing groups include multiple opaque objects. When drawing opaque objects in multiple drawing groups based on multiple stencil values, the processing device can utilize a stencil buffer to draw the multiple opaque objects in a visually forward-to-backward order. The processing device can update the stencil buffer based on the position of the drawn opaque object and the stencil value of the corresponding drawing group while drawing multiple opaque objects. In this way, the processing device can utilize the stencil buffer and draw multiple opaque objects in a forward-to-backward order, allowing foreground objects to be written into the stencil information first and occlude background objects, improving the accuracy of occlusion culling. Furthermore, this method can reduce redundant drawing and calculation of redundant pixels, thereby improving drawing efficiency.

[0043] In some embodiments, multiple drawing groups include multiple transparent or semi-transparent objects. The processing device can utilize a stencil buffer to draw multiple transparent or semi-transparent objects in a visually forward-to-backward order. In this way, the processing device can draw transparent or semi-transparent objects in a forward-to-backward order after the opaque objects have been drawn, which ensures proper blending of transparent pixels with the background and reduces blending errors.

[0044] Figure 4 A schematic diagram of example 400, illustrating the adjustment of the drawing order of multiple objects in multiple drawing groups according to some embodiments of the present disclosure, is shown. Figure 4 As shown, Example 400 includes drawing groups A, B, and C, where drawing groups A and C each include opaque objects, semi-transparent objects, and transparent objects, and drawing group B includes semi-transparent objects and transparent objects. Each drawing group can represent an application window, a system pop-up, an interactive component, or a background, etc. For example, drawing group A could be a window of an instant messaging application, drawing group B could be a window of a browser application, and drawing group C could be a system pop-up. In Example 400, drawing group A is the drawing group that is visually furthest from the user (e.g., lowest in hierarchy), drawing group C is the drawing group that is visually closest to the user (e.g., highest in hierarchy), and drawing group B is the drawing group that is visually between drawing groups A and drawing group C.

[0045] In related technologies, the processing device can draw A, B, and C in a visually forward-to-backward order. For example, the processing device can first draw the semi-transparent object, the opaque object, and the transparent object in A, then draw the semi-transparent object and the transparent object in B, and finally draw the semi-transparent object, the opaque object, and the transparent object in C.

[0046] When using the stencil buffer in the embodiments of this disclosure to optimize the drawing process, the processing device can distinguish between opaque objects and transparent or semi-transparent objects in multiple drawing groups. The processing device can sort the opaque objects in the multiple drawing groups in a visually forward-to-backward order, and sort the transparent and semi-transparent objects in the multiple drawing groups in a visually backward-to-backward order. Then, the processing device can assign a unique, decreasing stencil value to each drawing group in a visually forward-to-backward order. For example, the nearest drawing group C can be assigned a stencil value of 127, drawing group B can be assigned a stencil value of 126, and the farthest drawing group A can be assigned a stencil value of 125. The processing device can then draw the sorted opaque objects first based on the assigned stencil values. During the drawing of opaque objects, the processing device can update the stencil buffer based on the stencil values ​​of the drawn opaque objects. After all opaque objects have been drawn, the processing device can draw the sorted transparent and semi-transparent objects based on the assigned stencil values. Since transparent and semi-transparent objects do not obscure other objects, the processing device does not update the stencil buffer when drawing transparent and semi-transparent objects, thus improving visual drawing accuracy.

[0047] like Figure 4 As shown, in the adjusted drawing order, all opaque objects in multiple drawing groups are drawn first, following a visual order from near to far; that is, the opaque objects in group C are drawn first, followed by those in group A. After all opaque objects have been drawn, all transparent or semi-transparent objects in multiple drawing groups are drawn next, following a visual order from far to near; that is, the transparent or semi-transparent objects in group A are drawn first, followed by those in group B, and finally those in group C.

[0048] By drawing opaque objects in a near-to-far order and simultaneously updating the stencil buffer, occlusion relationships can be determined based on the stencil values ​​of already drawn objects when drawing lower-level objects, thus eliminating occluded pixels and improving drawing efficiency. Drawing transparent and semi-transparent objects in a far-to-near order allows for precise control over the blending of pixels from transparent and semi-transparent objects with pixels from underlying objects, improving drawing accuracy.

[0049] In some embodiments, the layer information of the first object is first layer information. When updating the stencil buffer corresponding to the drawing space based on the layer information of the first object, the processing device can obtain second layer information at the position corresponding to the first pixel in the stencil buffer when drawing the first pixel of the first object. The processing device can determine that the first pixel is not occluded based on the first layer information and the second layer information. The processing device can update the layer information at the position corresponding to the first pixel in the stencil buffer to the first layer information. In this way, it can be ensured that the layer information in the stencil buffer always reflects the latest visible layer, thereby further improving the accuracy and stability of occlusion judgment.

[0050] In some embodiments, the plurality of opaque objects includes a third object. When drawing multiple opaque objects in a visually distant-to-near order using a stencil buffer, the processing device can obtain third-level information of the corresponding drawing group of the third object. The processing device can obtain fourth-level information at the position corresponding to the second pixel in the stencil buffer when drawing the second pixel of the third object. The processing device can determine that the second pixel is not occluded based on the third-level and fourth-level information. Then, the processing device can draw the third object by blending the second pixel with the third pixel corresponding to the second pixel in the drawing space. In this way, the overlay drawing of transparent and semi-transparent objects can be achieved. This method can precisely control the occlusion and blending relationship between transparent or semi-transparent objects and opaque objects, reducing erroneous occlusion between transparent layers.

[0051] Figure 5 A schematic diagram of example 500, illustrating the drawing of multiple objects in multiple drawing groups according to some embodiments of the present disclosure, is shown. Example 500 includes a stencil buffer 502. At step 1, the processing device may initialize the stencil buffer to initialize the values ​​in the stencil buffer to original stencil values, such as 0. For example, assuming the resolution of the drawing space is 1920 × 1080 pixels, the stencil buffer 502 may include 1920 × 1080 stencil values, each of which may be 8 bits and initialized to 0.

[0052] In step 2, the processing device can draw the opaque object in the visually nearest drawing group C and update the stencil buffer based on the stencil values ​​assigned to drawing group C. For example... Figure 5 As shown, in step 2, the template values ​​in the template buffer corresponding to the pixels of the opaque objects in drawing group C are all updated to the template value 127 of drawing group C.

[0053] In step 3, the processing device can draw the opaque objects in group A. The processing device can pre-set the pass conditions for the stencil test; for example, the pass condition can be set to the stencil value of the drawn pixel being greater than or equal to the corresponding stencil value in the stencil buffer. In other words, if the stencil value of the drawn pixel is greater than or equal to the corresponding stencil value in the stencil buffer, the pixel passes the stencil test and is drawn, while the corresponding stencil value in the stencil buffer is updated to the stencil value of the pixel. If the stencil value of the drawn pixel is less than the corresponding stencil value in the stencil buffer, the pixel fails the stencil test and is discarded. Discarded pixels will not participate in the drawing; that is, the processing device will not calculate the drawing information for that pixel. In Example 500, the stencil value of drawing group A is 125. Therefore, when comparing the lower right portion of the opaque objects in drawing group A with the corresponding stencil value in the stencil buffer, since 125 is less than 127, these pixels will be discarded and will not participate in the drawing. When the other parts of the opaque object in drawing group A are compared with the corresponding stencil value in the stencil buffer, these pixels will be drawn because 125 is greater than 0, and the corresponding stencil value in the stencil buffer will be updated from 0 to 125.

[0054] In step 4, after all opaque objects have been drawn, the processing device can draw transparent and semi-transparent objects in visually forward order. In Example 500, taking the drawing of transparent objects in group B as an example, the stencil value of group B is 126. The processing device can compare the stencil values ​​of the pixels of the transparent objects with the corresponding stencil values ​​in the stencil buffer. When comparing the lower right portion of the transparent objects in group B with the corresponding stencil value in the stencil buffer, these pixels are discarded and not drawn because 126 is less than 127. When comparing the upper left portion of the transparent objects in group B with the corresponding stencil value in the stencil buffer, these pixels are blended with the opaque objects in group A because 126 is greater than 125. Furthermore, since transparent objects are being drawn, the corresponding stencil values ​​in the stencil buffer are not updated.

[0055] In this way, stencil testing achieves better occlusion culling efficiency than depth testing or LRZ testing, and GPU performance can be optimized by approximately 6-10%. Furthermore, this approach can meet the needs of Spatial OS's drawing space for real-time dynamic changes in object depth.

[0056] In some applications, it's desirable for drawn objects to have a feathering effect. Feathering refers to applying a gradual change in transparency to the edges of an object, making the edges transition from completely opaque to completely transparent, thus eliminating hard edges and allowing the object to blend naturally with the background. This process can be viewed as overlaying a feathered object onto the feathered object to achieve the feathering effect.

[0057] In some embodiments, in response to the first object including opaque, semi-transparent, and transparent portions, the processing device can acquire transparency information of the first object. The processing device can assign multiple levels of layer information to multiple portions of the first object based on the transparency information. In this way, by using transparency information to assign different levels of layer information to different portions of the effect object, dynamic rendering control of special materials such as feathering, glass, and gradient edges can be achieved.

[0058] In some embodiments, when assigning multiple levels of information to multiple parts of a first object based on the transparency information of the first object, the processing device may assign a first template value to the opaque parts of the first object, the first template value indicating that the corresponding position in the drawing space is occluded. The processing device may assign second template values ​​to the semi-transparent and transparent parts of the first object, the second template value indicating that the corresponding position in the drawing space is not occluded. In this way, the gradient effect of the transparent area can be preserved while ensuring the correctness of occlusion, thereby improving the efficiency of drawing while ensuring the accuracy of drawing.

[0059] Figure 6 A schematic diagram of an example 600 for drawing a feathered object according to some embodiments of the present disclosure is shown. Figure 6 As shown, the processing device can identify the visible and invisible areas of the feathered object in the eye buffer, where the central area of ​​the eye buffer is the visible area and the surrounding annular area is the invisible area, and the transparency of the visible area can gradually transition from complete transparency at the center to complete opacity at the edge.

[0060] In step 1, the processing device can acquire an alpha map of the feathered object. In the alpha map, an alpha value of 1 represents complete opacity, an alpha value less than 1 and greater than 0 represents semi-transparency with a gradual change in transparency within this area. An alpha value of 0 represents complete opacity, meaning that the object below is obscured.

[0061] In step 2, the processing device can update the stencil value in the stencil buffer based on the transparency map of the feathered object. For example, when the transparency of a pixel is equal to 1, the stencil value corresponding to that pixel in the stencil buffer can be set to 0. When the transparency of a pixel is less than 1, the stencil value corresponding to that pixel in the stencil buffer can be set to 1.

[0062] In step 3, after updating the stencil buffer based on the alpha map, the processing device can perform a stencil test on objects behind the feathered object based on the updated stencil buffer. In Example 600, the processing device can set the pass condition for the stencil test to a stencil value of 1 in the stencil buffer. The processing device can discard pixels that fail the stencil test so that they are not involved in the drawing calculation. The processing device can retain pixels that pass the stencil test and draw those pixels based on the alpha of the feathered object. In this way, pixels behind the invisible area in the eye buffer can be discarded, while pixels behind the visible area can be retained and drawn. In this way, the processing device can discard pixels behind the invisible area of ​​the feathered object based on hierarchical information, thereby improving the drawing efficiency of the feathering effect.

[0063] Figure 7 A block diagram of an apparatus 700 for drawing objects according to some embodiments of the present disclosure is shown. Figure 7 As shown, the device 700 includes a first object drawing module 702, configured to draw a first object in the drawing space of an extended reality environment, wherein at least a portion of the first object is opaque. The device 700 also includes an occlusion portion determination module 704, configured to determine a first portion of a second object occluded by the first object based on the layer information of the first object. Furthermore, the device 700 includes a second object drawing module 706, configured to draw a second portion of the second object that is not occluded by the first object, provided that the first portion of the second object is not involved in the drawing.

[0064] In some embodiments, the occlusion determination module 704 includes: a template buffer update module configured to update a template buffer corresponding to the drawing space based on the hierarchical information of the first object, the template buffer containing hierarchical information of at least a portion of the pixels in the drawing space; and a template buffer usage module configured to determine a first portion of the second object occluded by the first object based on the updated template buffer.

[0065] In some embodiments, the layer information of the first object is first layer information, and the template buffer update module includes: a second layer information acquisition module, configured to acquire second layer information at the position corresponding to the first pixel in the template buffer when drawing the first pixel of the first object; a second layer information usage module, configured to determine that the first pixel is not occluded based on the first layer information and the second layer information; and update the layer information at the position corresponding to the first pixel in the template buffer to the first layer information.

[0066] In some embodiments, the hierarchy information is template values, and the apparatus 700 further includes: a template value allocation module configured to allocate multiple template values ​​to multiple drawing groups based on the visual hierarchy of the multiple drawing groups, wherein a first drawing group in the multiple drawing groups includes a first object, and a second drawing group in the multiple drawing groups includes a second object; a first opaque object drawing module configured to draw opaque objects in the multiple drawing groups based on the multiple template values; and a first transparent object drawing module configured to draw transparent or semi-transparent objects in the multiple drawing groups after all opaque objects in the multiple drawing groups have been drawn.

[0067] In some embodiments, the plurality of drawing groups include a plurality of opaque objects, and drawing the opaque objects in the plurality of drawing groups based on a plurality of template values ​​includes: a second opaque object drawing module configured to draw the plurality of opaque objects in a visual order from near to far using a template buffer; and a first template buffer updating module configured to update the template buffer based on the position of the drawn opaque objects and the template value of the corresponding drawing group when drawing the plurality of opaque objects.

[0068] In some embodiments, the plurality of drawing groups include a plurality of transparent or semi-transparent objects, and the first transparent object drawing module includes: a second transparent object drawing module configured to draw a plurality of transparent or semi-transparent objects in a visually distant to near order using a stencil buffer.

[0069] In some embodiments, the plurality of opaque objects include a third object, and the second opaque object drawing module includes: a third-level information acquisition module configured to acquire third-level information of the corresponding drawing group of the third object; a fourth-level information acquisition module configured to acquire fourth-level information at the position corresponding to the second pixel in the template buffer when drawing the second pixel of the third object; a pixel occlusion determination module configured to determine that the second pixel is not occluded based on the third-level information and the fourth-level information; and a pixel blending module configured to draw the third object by blending the second pixel and the third pixel corresponding to the second pixel in the drawing space.

[0070] In some embodiments, the drawing group among the plurality of drawing groups includes an application window, a system pop-up, or an interactive component.

[0071] In some embodiments, the apparatus 700 further includes: a transparency information acquisition module configured to acquire transparency information of the first object in response to the first object including an opaque portion, a semi-transparent portion, and a transparent portion; and a transparency information usage allocation module configured to allocate multiple levels of information to multiple portions of the first object based on the transparency information of the first object.

[0072] In some embodiments, the transparency information usage module includes: a first template value allocation module configured to allocate a first template value to the opaque portion of the first object, the first template value indicating that the corresponding position in the drawing space is occluded; and a second template value allocation module configured to allocate a second template value to the semi-transparent and transparent portions of the first object, the second template value indicating that the corresponding position in the drawing space is not occluded.

[0073] It is understood that by utilizing the apparatus 700 of this disclosure, at least one of the many advantages achievable by the methods or processes described above can be realized. For example, by using hierarchical information instead of depth values ​​to determine the occlusion relationship between objects, pixels occluded by the first object can be removed when drawing the second object, thereby reducing the processor's calculation of drawing information for invisible pixels, reducing the amount of computation of invalid drawing information, improving the efficiency of the drawing process, and reducing the waste of processing resources. Furthermore, using hierarchical information to determine the occlusion relationship between objects can correctly determine the occlusion area even when the depth of the object dynamically changes in scenarios such as multi-window overlay or dynamic user interaction panels, without having to recalculate the depth value of the pixel, thereby reducing the waste of processing resources.

[0074] Figure 8 A block diagram of a device 800 capable of implementing various embodiments of the present disclosure is shown. Device 800 may be, for example, as... Figure 1 The processing device 102 shown. For example... Figure 8 As shown, device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the operation of device 800. The CPU / GPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804. Although not shown in... Figure 8 As shown, device 800 may also include a coprocessor.

[0075] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0076] The various methods or processes described above can be executed by CPU / GPU 801. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU / GPU 801, one or more steps or actions in the methods or processes described above can be performed.

[0077] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0078] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0079] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0080] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to execute the computer-readable program instructions, thereby implementing various aspects of this disclosure.

[0081] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0082] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0084] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for drawing objects, comprising: Draw a first object in the drawing space of the extended reality environment, wherein at least a portion of the first object is opaque; Based on the hierarchical information of the first object, determine the first part of the second object that is occluded by the first object; as well as If the first part of the second object is not involved in the drawing, draw the second part of the second object that is not obscured by the first object.

2. The method of claim 1, wherein determining the first portion of the second object occluded by the first object based on the hierarchy information of the first object comprises: The template buffer corresponding to the drawing space is updated based on the hierarchical information of the first object, and the template buffer contains hierarchical information of at least a portion of the pixels in the drawing space; as well as The first portion of the second object that is occluded by the first object is determined based on the updated template buffer.

3. The method according to claim 2, wherein the hierarchy information of the first object is first hierarchy information, and updating the stencil buffer corresponding to the drawing space based on the hierarchy information of the first object comprises: When drawing the first pixel of the first object, the second-level information at the position corresponding to the first pixel in the template buffer is obtained; Based on the first-level information and the second-level information, it is determined that the first pixel is not occluded; as well as Update the hierarchical information at the position corresponding to the first pixel in the template buffer to the first hierarchical information.

4. The method according to claim 2, wherein the hierarchy information is a template value, and the method further comprises: Multiple template values ​​are assigned to the multiple drawing groups based on their visual hierarchical relationship, wherein the first drawing group in the multiple drawing groups includes the first object, and the second drawing group in the multiple drawing groups includes the second object; Based on the multiple template values, draw the opaque objects in the multiple drawing groups; as well as After all the opaque objects in the plurality of drawing groups have been drawn, the transparent or semi-transparent objects in the plurality of drawing groups are drawn.

5. The method of claim 4, wherein the plurality of drawing groups comprises a plurality of opaque objects, and drawing the opaque objects in the plurality of drawing groups based on the plurality of template values ​​comprises: Using the template buffer, the multiple opaque objects are drawn in visual order from near to far; as well as When drawing the plurality of opaque objects, the template buffer is updated based on the position of the drawn opaque object and the template value of the corresponding drawing group.

6. The method of claim 5, wherein the plurality of drawing groups comprises a plurality of transparent or semi-transparent objects, and drawing the transparent objects in the plurality of drawing groups after all opaque objects in the plurality of drawing groups have been drawn comprises: Using the template buffer, the multiple transparent or semi-transparent objects are drawn in visual order from far to near.

7. The method of claim 6, wherein the plurality of opaque objects includes a third object, and drawing the plurality of opaque objects in a visually distant-to-near order using the stencil buffer comprises: Obtain the third-level information of the corresponding drawing group of the third object; When drawing the second pixel of the third object, obtain the fourth-level information at the position corresponding to the second pixel in the template buffer; Based on the third-level information and the fourth-level information, it is determined that the second pixel is not occluded; as well as The third object is drawn by mixing the second pixel with the third pixel in the drawing space that corresponds to the second pixel.

8. The method according to claim 4, wherein the drawing group in the plurality of drawing groups includes an application window, a system pop-up, or an interactive component.

9. The method according to claim 1, further comprising: In response to the first object including an opaque portion, a semi-transparent portion, and a transparent portion, the transparency information of the first object is obtained; as well as Multiple layers of information are assigned to multiple parts of the first object based on the transparency information of the first object.

10. The method of claim 9, wherein assigning the plurality of hierarchical information to the plurality of portions of the first object based on the transparency information of the first object comprises: A first template value is assigned to the opaque portion of the first object, the first template value indicating that the corresponding position in the drawing space is occluded; as well as A second template value is assigned to the semi-transparent portion and the transparent portion of the first object, the second template value indicating that the corresponding position in the drawing space is not occluded.

11. An apparatus for drawing objects, comprising: A first object drawing module is configured to draw a first object in the drawing space of an extended reality environment, wherein at least a portion of the first object is opaque; The occlusion determination module is configured to determine the first part of the second object that is occluded by the first object based on the hierarchical information of the first object; as well as The second object drawing module is configured to draw the second part of the second object that is not occluded by the first object, when the first part of the second object is not involved in the drawing.

12. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 10.

13. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 10.

Images